[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:49 dist-test-slave-scsf krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:49 dist-test-slave-scsf krb5kdc[399](info): set up 2 sockets
Jun 20 08:59:49 dist-test-slave-scsf krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 08:59:50.139454 392 mini_kdc.cc:121] Time spent starting KDC: real 2.399s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 08:59:50.328392 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.188s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 08:59:50.502837 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.174s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/test-user.keytab.
W20260620 08:59:51.523826 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.931s user 0.000s sys 0.002s
Jun 20 08:59:51 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 08:59:51.714879 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.168s user 0.000s sys 0.003s
I20260620 08:59:51.715783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:51.828745 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:51.829070 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:51.829166 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:51.833267 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:51.833349 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:51.833387 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:51.833444 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:51.833487 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:51.838977 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:51.840137 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:51.841297 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:51.850271 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:51.850260 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:51.850260 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:51.850523 412 server_base.cc:1034] running on GCE node
I20260620 08:59:51.851198 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:51.852312 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:51.853471 412 hybrid_clock.cc:648] HybridClock initialized: now 1781945991853420 us; error 69 us; skew 500 ppm
Jun 20 08:59:51 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:51.856359 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:51.857331 412 webserver.cc:460] Webserver started at http://127.0.98.62:38449/ using document root <none> and password file <none>
I20260620 08:59:51.857874 412 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:51.857930 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:51.858110 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:51.860150 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "b4183faab1604ec19ac0523472dc17e0"
format_stamp: "Formatted at 2026-06-20 08:59:51 on dist-test-slave-scsf"
server_key: "f3bd0d0df36c76d5400eefeac2baaab2"
server_key_iv: "43d9eedd336ed54660f579bcb99cb4ec"
server_key_version: "encryptionkey@0"
I20260620 08:59:51.860728 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4183faab1604ec19ac0523472dc17e0"
format_stamp: "Formatted at 2026-06-20 08:59:51 on dist-test-slave-scsf"
server_key: "f3bd0d0df36c76d5400eefeac2baaab2"
server_key_iv: "43d9eedd336ed54660f579bcb99cb4ec"
server_key_version: "encryptionkey@0"
I20260620 08:59:51.864533 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 08:59:51.866957 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:51.868186 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:59:51.868338 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "b4183faab1604ec19ac0523472dc17e0"
format_stamp: "Formatted at 2026-06-20 08:59:51 on dist-test-slave-scsf"
server_key: "f3bd0d0df36c76d5400eefeac2baaab2"
server_key_iv: "43d9eedd336ed54660f579bcb99cb4ec"
server_key_version: "encryptionkey@0"
I20260620 08:59:51.868451 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:51.886502 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:51.889945 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:51.890172 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:51.897377 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43227
I20260620 08:59:51.897401 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43227 every 8 connection(s)
I20260620 08:59:51.898404 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:51.901314 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:51.901917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 412
I20260620 08:59:51.902088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:51.903348 392 external_mini_cluster.cc:1428] Setting key d9972727d9465cff6a24c5c0e8908098
I20260620 08:59:51.907598 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0: Bootstrap starting.
I20260620 08:59:51.909904 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:51.910631 480 log.cc:824] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:51.912580 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0: No bootstrap required, opened a new log
I20260620 08:59:51.916231 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } }
I20260620 08:59:51.916491 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:51.916574 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4183faab1604ec19ac0523472dc17e0, State: Initialized, Role: FOLLOWER
I20260620 08:59:51.917096 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } }
I20260620 08:59:51.917255 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:51.917320 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:51.917414 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:51.918475 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } }
I20260620 08:59:51.918840 480 leader_election.cc:302] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4183faab1604ec19ac0523472dc17e0; no voters:
I20260620 08:59:51.919221 480 leader_election.cc:288] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:51.919323 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:51.919570 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [term 1 LEADER]: Becoming Leader. State: Replica: b4183faab1604ec19ac0523472dc17e0, State: Running, Role: LEADER
I20260620 08:59:51.919939 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } }
I20260620 08:59:51.920567 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 08:59:51 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:51.921684 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4183faab1604ec19ac0523472dc17e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } } }
I20260620 08:59:51.921805 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:51.922147 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4183faab1604ec19ac0523472dc17e0. Latest consensus state: current_term: 1 leader_uuid: "b4183faab1604ec19ac0523472dc17e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4183faab1604ec19ac0523472dc17e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43227 } } }
I20260620 08:59:51.922236 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:51.923069 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:51.925678 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:51.928928 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:51.906469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.0.98.62:43227)
0620 08:59:51.906886 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:51.906901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:51.908024 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 08:59:51.911223 (+ 3199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:51.911239 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:51.911535 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:51.911857 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:51.914120 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:51.915527 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:51.916881 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:51.917591 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:51.917766 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:51.922841 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:51.922847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:51.922864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:51.922895 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:51.925029 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:51.925647 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:51.925654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:51.925660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:51.925868 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:51.926287 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:51.926290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:51.926292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:51.926780 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:51.926977 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:51.927639 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 08:59:51.927936 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":144,"threads_started":1}
I20260620 08:59:51.931615 494 catalog_manager.cc:1347] Generated new cluster ID: 255f122fe4b04081ae4b7646ced6432a
I20260620 08:59:51.931686 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:51.946282 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:51.947427 494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:51.951287 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4183faab1604ec19ac0523472dc17e0: Generated new TSK 0
I20260620 08:59:51.951994 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:59:51.990913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:52.113190 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:52.113451 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:52.113502 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:52.117564 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:52.117625 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:52.117684 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:59:52.123126 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:52.124118 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:52.125180 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:52.134315 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:52.134428 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:52.134447 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:52.134675 506 server_base.cc:1034] running on GCE node
I20260620 08:59:52.135026 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:52.135603 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:52.136754 506 hybrid_clock.cc:648] HybridClock initialized: now 1781945992136741 us; error 24 us; skew 500 ppm
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:52.139271 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:59:52.140081 506 webserver.cc:460] Webserver started at http://127.0.98.1:40189/ using document root <none> and password file <none>
I20260620 08:59:52.140650 506 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:52.140712 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:52.140872 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:52.142796 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "91c45884770a4e90a37b70a0ab76651d"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "06f9ba79f8225f5eb0b1803f30564974"
server_key_iv: "484164aa2303bc85dffcbf6e549f947c"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.143314 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91c45884770a4e90a37b70a0ab76651d"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "06f9ba79f8225f5eb0b1803f30564974"
server_key_iv: "484164aa2303bc85dffcbf6e549f947c"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.146857 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 08:59:52.149339 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.150524 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:52.150665 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "91c45884770a4e90a37b70a0ab76651d"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "06f9ba79f8225f5eb0b1803f30564974"
server_key_iv: "484164aa2303bc85dffcbf6e549f947c"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.150776 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:52.170190 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:52.173280 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:52.173493 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:52.174094 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:52.174932 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:52.175015 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.175091 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:52.175137 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.184111 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35109
I20260620 08:59:52.184137 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35109 every 8 connection(s)
I20260620 08:59:52.185058 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:59:52.187867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 506
I20260620 08:59:52.187968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:59:52.188205 392 external_mini_cluster.cc:1428] Setting key 2cd39053d20875749a9baa151a7c635e
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:59:52.199445 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.186778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46875 (local address 127.0.98.62:43227)
0620 08:59:52.186918 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:52.186922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:52.187701 (+ 779us) server_negotiation.cc:407] Connection header received
0620 08:59:52.189278 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:52.189281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:52.189323 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:52.189394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:52.190642 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.191456 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.193022 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.193266 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.193379 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.195803 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:52.195811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:52.195813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:52.195835 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:52.197378 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.197911 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.197915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.197917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.198011 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.198353 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.198358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.198360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.198539 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:52.198646 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:52.199132 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 08:59:52.199267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 08:59:52.200423 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.187104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43227 (local address 127.0.98.1:46875)
0620 08:59:52.187549 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:52.187590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:52.189062 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:52.189596 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:52.189606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:52.190025 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:52.190491 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.190504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.191585 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.191591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.192873 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.192883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.193418 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.193422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.193647 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.194253 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:52.194268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:52.195640 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:52.197529 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.197534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.197550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.197803 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.198112 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.198114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.198116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.198258 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.198643 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:52.198650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:52.198895 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 08:59:52.199081 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 08:59:52.199399 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260620 08:59:52.202052 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43227
I20260620 08:59:52.202337 635 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:52.202921 635 heartbeater.cc:511] Master 127.0.98.62:43227 requested a full tablet report, sending...
I20260620 08:59:52.204588 444 ts_manager.cc:194] Registered new tserver with Master: 91c45884770a4e90a37b70a0ab76651d (127.0.98.1:35109)
I20260620 08:59:52.205771 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:59:52.215368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:52.326148 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:52.326426 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:52.326471 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:52.330546 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:52.330632 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:52.330742 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:59:52.336186 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:52.337571 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:52.338634 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:52.348333 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:52.348331 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:52.348508 640 server_base.cc:1034] running on GCE node
W20260620 08:59:52.348326 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:52.349092 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:52.349742 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:52.350972 640 hybrid_clock.cc:648] HybridClock initialized: now 1781945992350955 us; error 27 us; skew 500 ppm
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:52.353874 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:59:52.354802 640 webserver.cc:460] Webserver started at http://127.0.98.2:33473/ using document root <none> and password file <none>
I20260620 08:59:52.355316 640 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:52.355373 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:52.355547 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:52.357571 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "728f3063c8d3443e9553d90dfdd6a0d8"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "52d93f09d32a1a8d7e9e62e310aabd69"
server_key_iv: "4ccfb2779c68178fdec9a2b96460aa1b"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.358083 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "728f3063c8d3443e9553d90dfdd6a0d8"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "52d93f09d32a1a8d7e9e62e310aabd69"
server_key_iv: "4ccfb2779c68178fdec9a2b96460aa1b"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.361743 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:59:52.364095 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.365352 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:52.365489 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "728f3063c8d3443e9553d90dfdd6a0d8"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "52d93f09d32a1a8d7e9e62e310aabd69"
server_key_iv: "4ccfb2779c68178fdec9a2b96460aa1b"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.365602 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:52.376608 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:52.379899 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:52.380105 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:52.380782 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:52.381604 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:52.381649 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.381716 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:52.381752 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.390950 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37061
I20260620 08:59:52.390980 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37061 every 8 connection(s)
I20260620 08:59:52.391820 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:59:52.401285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 640
I20260620 08:59:52.401415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:52.401721 392 external_mini_cluster.cc:1428] Setting key 78f31523f90030a754b448c93a809743
I20260620 08:59:52.406294 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.393638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50317 (local address 127.0.98.62:43227)
0620 08:59:52.393782 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:52.393791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:52.394609 (+ 818us) server_negotiation.cc:407] Connection header received
0620 08:59:52.395905 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:52.395909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:52.395959 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:52.396040 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:52.397340 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.397876 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.398958 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.399167 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.399243 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.402248 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:52.402253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:52.402255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:52.402287 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:52.403841 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.404714 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.404718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.404720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.404809 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.405182 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.405185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.405187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.405379 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:52.405489 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:52.405994 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 08:59:52.406129 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 08:59:52.407090 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.394030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43227 (local address 127.0.98.2:50317)
0620 08:59:52.394454 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:52.394497 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:52.395689 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:52.396210 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:52.396217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:52.396663 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:52.397163 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.397177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.397994 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.397997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.398849 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.398857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.399291 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.399293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.399435 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.400002 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:52.400177 (+ 175us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:52.402039 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:52.403959 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.403976 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.403993 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.404364 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.404915 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.404919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.404921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.405069 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.405490 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:52.405497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:52.405745 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 08:59:52.405932 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 08:59:52.406255 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260620 08:59:52.408354 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43227
I20260620 08:59:52.408638 769 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:52.409263 769 heartbeater.cc:511] Master 127.0.98.62:43227 requested a full tablet report, sending...
I20260620 08:59:52.410493 444 ts_manager.cc:194] Registered new tserver with Master: 728f3063c8d3443e9553d90dfdd6a0d8 (127.0.98.2:37061)
I20260620 08:59:52.411309 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50317
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:59:52.429679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:52.539544 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:52.539793 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:52.539835 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:52.543711 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:52.543761 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:52.543817 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:59:52.549034 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:52.550048 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:52.550947 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:52.560434 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:52.560425 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:52.560405 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:52.560667 774 server_base.cc:1034] running on GCE node
I20260620 08:59:52.561156 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:52.561704 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:52.562860 774 hybrid_clock.cc:648] HybridClock initialized: now 1781945992562835 us; error 37 us; skew 500 ppm
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:52.565449 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:59:52.566278 774 webserver.cc:460] Webserver started at http://127.0.98.3:35293/ using document root <none> and password file <none>
I20260620 08:59:52.566817 774 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:52.566874 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:52.567126 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:52.569211 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a816bdb6b23403496707b336817cc66"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "8b69ee7b9453957677e0f781a96339e6"
server_key_iv: "b31972c811da2748e7f10fb16f8c853a"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.569837 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a816bdb6b23403496707b336817cc66"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "8b69ee7b9453957677e0f781a96339e6"
server_key_iv: "b31972c811da2748e7f10fb16f8c853a"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.573652 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 08:59:52.575838 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.577068 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:59:52.577190 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "4a816bdb6b23403496707b336817cc66"
format_stamp: "Formatted at 2026-06-20 08:59:52 on dist-test-slave-scsf"
server_key: "8b69ee7b9453957677e0f781a96339e6"
server_key_iv: "b31972c811da2748e7f10fb16f8c853a"
server_key_version: "encryptionkey@0"
I20260620 08:59:52.577291 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:52.606927 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:52.609920 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:52.610138 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:52.611155 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:52.611963 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:52.612038 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.612133 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:52.612183 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:52.622037 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34553
I20260620 08:59:52.622083 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34553 every 8 connection(s)
I20260620 08:59:52.622893 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:59:52.625530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 774
I20260620 08:59:52.625643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:59:52.625916 392 external_mini_cluster.cc:1428] Setting key a143c451be79bf5c5dcaddab834913cc
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:52.636075 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.624647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52113 (local address 127.0.98.62:43227)
0620 08:59:52.624781 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:52.624785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:52.625727 (+ 942us) server_negotiation.cc:407] Connection header received
0620 08:59:52.627398 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:52.627400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:52.627441 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:52.627501 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:52.628788 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.629303 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.630360 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.630568 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.630641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.632943 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:52.632946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:52.632948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:52.632970 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:52.634218 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.634684 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.634687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.634689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.634801 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.635153 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.635156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.635157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.635352 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:52.635428 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:52.635844 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 08:59:52.635961 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:59:52.636850 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.625057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43227 (local address 127.0.98.3:52113)
0620 08:59:52.625546 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:52.625607 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:52.627168 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:52.627658 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:52.627664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:52.628065 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:52.628670 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.628684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.629400 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.629402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.630264 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:52.630272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.630683 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.630686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:52.630814 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.631428 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:52.631444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:52.632805 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:52.634317 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.634321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.634332 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.634573 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.634890 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:52.634892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:52.634893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:52.635029 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:52.635433 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:52.635437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:52.635644 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 08:59:52.635784 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 08:59:52.636028 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260620 08:59:52.637982 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43227
I20260620 08:59:52.638175 903 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:52.638619 903 heartbeater.cc:511] Master 127.0.98.62:43227 requested a full tablet report, sending...
I20260620 08:59:52.639772 444 ts_manager.cc:194] Registered new tserver with Master: 4a816bdb6b23403496707b336817cc66 (127.0.98.3:34553)
I20260620 08:59:52.640195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:59:52.640420 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52113
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:52.666524 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.657268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.0.98.1:35109)
0620 08:59:52.657583 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:52.657596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:52.657620 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:59:52.657730 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:52.657736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:52.657934 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:52.658065 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:52.658737 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.659511 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.660282 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.660701 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.660787 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.663015 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:52.663023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:52.663035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:52.663058 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:52.664733 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.665166 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.665171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.665172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.665250 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.665527 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.665531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.665533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.665768 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:52.666008 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:52.666190 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 08:59:52.666338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
W20260620 08:59:52.667416 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59716
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:52.692288 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:52.684438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.0.98.1:35109)
0620 08:59:52.684618 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:52.684621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:52.684639 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:59:52.684767 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:52.684769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:52.684809 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:52.684875 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:52.685514 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.686072 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.686880 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:52.687130 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:52.687205 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:52.689502 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:52.689505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:52.689507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:52.689528 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:52.690934 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.691305 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.691308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.691309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.691385 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:52.691599 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:52.691601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:52.691602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:52.691743 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:52.691833 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:52.692049 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:59:52.692138 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:59:52.693612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 506
I20260620 08:59:52.701205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 640
I20260620 08:59:52.708899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 774
I20260620 08:59:52.714140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 412
2026-06-20T08:59:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5005 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[917](info): set up 2 sockets
Jun 20 08:59:52 dist-test-slave-scsf krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 08:59:53.752182 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:53 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:53.865474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37413
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:53.971601 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:53.971900 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:53.971971 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:53.975898 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:53.975992 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:53.976022 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:53.976050 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:53.976085 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:53.981259 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:53.982278 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:53.983289 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:53.991410 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:53.991410 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:53.991554 930 server_base.cc:1034] running on GCE node
W20260620 08:59:53.991441 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:53.992338 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:53.993419 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:53.994573 930 hybrid_clock.cc:648] HybridClock initialized: now 1781945993994556 us; error 30 us; skew 500 ppm
Jun 20 08:59:53 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:54.000015 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:54.000954 930 webserver.cc:460] Webserver started at http://127.0.98.62:38547/ using document root <none> and password file <none>
I20260620 08:59:54.001564 930 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:54.001657 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:54.001888 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:54.003903 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "d537122c7956421595524973563f4cbf"
format_stamp: "Formatted at 2026-06-20 08:59:53 on dist-test-slave-scsf"
server_key: "6f2ce9116707d8abc1145e494a353c8f"
server_key_iv: "a91362fd04ca17398fb65b485cf7b6fe"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.004528 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "d537122c7956421595524973563f4cbf"
format_stamp: "Formatted at 2026-06-20 08:59:53 on dist-test-slave-scsf"
server_key: "6f2ce9116707d8abc1145e494a353c8f"
server_key_iv: "a91362fd04ca17398fb65b485cf7b6fe"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.008067 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 08:59:54.010548 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.011844 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:54.011981 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "d537122c7956421595524973563f4cbf"
format_stamp: "Formatted at 2026-06-20 08:59:53 on dist-test-slave-scsf"
server_key: "6f2ce9116707d8abc1145e494a353c8f"
server_key_iv: "a91362fd04ca17398fb65b485cf7b6fe"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.012096 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:54.036922 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:54.039770 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:54.039980 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:54.046888 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37413
I20260620 08:59:54.046912 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37413 every 8 connection(s)
I20260620 08:59:54.047813 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:54.050439 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:54.052078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 930
I20260620 08:59:54.052176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:54.052435 392 external_mini_cluster.cc:1428] Setting key 4506c33b4d2df281eb3e7463601f16a5
I20260620 08:59:54.056404 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf: Bootstrap starting.
I20260620 08:59:54.058477 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:54.059194 998 log.cc:824] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf: Log is configured to *not* fsync() on all Append() calls
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:54.061339 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf: No bootstrap required, opened a new log
I20260620 08:59:54.065507 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260620 08:59:54.065735 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:54.065791 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d537122c7956421595524973563f4cbf, State: Initialized, Role: FOLLOWER
I20260620 08:59:54.066248 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260620 08:59:54.066365 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:54.066388 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.053852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.0.98.62:37413)
0620 08:59:54.054278 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.054293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.054326 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:59:54.054940 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.054956 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.055284 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.055625 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.056397 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.057393 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.058220 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.058753 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.058920 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.061688 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.061692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.061705 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.061730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.063387 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.063883 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.063887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.063892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.064009 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.064283 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.064285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.064286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.064729 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.064877 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.065172 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.065509 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260620 08:59:54.066444 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:54.066538 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:54.067375 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260620 08:59:54.067736 998 leader_election.cc:302] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d537122c7956421595524973563f4cbf; no voters:
I20260620 08:59:54.068078 998 leader_election.cc:288] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:54.068202 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:54.068449 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [term 1 LEADER]: Becoming Leader. State: Replica: d537122c7956421595524973563f4cbf, State: Running, Role: LEADER
I20260620 08:59:54.068828 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260620 08:59:54.069307 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:54.069933 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d537122c7956421595524973563f4cbf. Latest consensus state: current_term: 1 leader_uuid: "d537122c7956421595524973563f4cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } } }
I20260620 08:59:54.070137 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:54.070403 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d537122c7956421595524973563f4cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537122c7956421595524973563f4cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } } }
I20260620 08:59:54.070494 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:54.070698 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:54.073639 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:54.079247 1012 catalog_manager.cc:1347] Generated new cluster ID: 086710cc81d340cda416d880a2ec9966
I20260620 08:59:54.079313 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:54.093699 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:54.095602 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:54.104977 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d537122c7956421595524973563f4cbf: Generated new TSK 0
I20260620 08:59:54.105862 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:59:54.143026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:54.252672 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:54.252912 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:54.252952 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:54.256844 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:54.256889 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:54.256945 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:59:54.262041 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:54.263007 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:54.263968 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:54.273684 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.273675 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.273684 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:54.273836 1024 server_base.cc:1034] running on GCE node
I20260620 08:59:54.274233 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:54.274943 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:54.276101 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781945994276065 us; error 51 us; skew 500 ppm
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:54.278764 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:59:54.279659 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35717/ using document root <none> and password file <none>
I20260620 08:59:54.280242 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:54.280334 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:54.280625 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:54.282660 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "9208741a27bd44c2aceabfc382169625"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "73057181f74b6a3e191ba295fc6ba0b6"
server_key_iv: "7f83d18749da162ad962d8bdfdd7e49d"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.283244 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9208741a27bd44c2aceabfc382169625"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "73057181f74b6a3e191ba295fc6ba0b6"
server_key_iv: "7f83d18749da162ad962d8bdfdd7e49d"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.287070 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 08:59:54.289551 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.290750 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:54.290884 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "9208741a27bd44c2aceabfc382169625"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "73057181f74b6a3e191ba295fc6ba0b6"
server_key_iv: "7f83d18749da162ad962d8bdfdd7e49d"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.290988 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:54.305166 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:54.308467 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:54.308692 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:54.309332 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:54.310168 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:54.310241 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.310303 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:54.310354 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.319491 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41493
I20260620 08:59:54.319540 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41493 every 8 connection(s)
I20260620 08:59:54.320402 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:59:54.328943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1024
I20260620 08:59:54.329057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:59:54.329378 392 external_mini_cluster.cc:1428] Setting key 592f5babdd614014333188bfd6418a9c
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:54.334244 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.322183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55581 (local address 127.0.98.62:37413)
0620 08:59:54.322311 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.322315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.323147 (+ 832us) server_negotiation.cc:407] Connection header received
0620 08:59:54.324406 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.324408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.324505 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.324589 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.325703 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.326223 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.327298 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.327610 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.327691 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.330303 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.330306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.330308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.330330 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.332043 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.332659 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.332666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.332668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.332755 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.333146 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.333148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.333149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.333324 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.333396 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.334032 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.334130 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:59:54.335605 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.322579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.1:55581)
0620 08:59:54.322997 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.323040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.324219 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.324751 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.324758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.325107 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:54.325577 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.325591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.326324 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.326326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.327183 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.327192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.327738 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.327740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.327893 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.328425 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:54.328441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:54.330148 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:54.332179 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.332184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.332197 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.332543 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.332851 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.332856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.332860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.333049 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.333420 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:54.333430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:54.333780 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.333992 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.334345 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260620 08:59:54.336934 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260620 08:59:54.337209 1153 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:54.337810 1153 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 08:59:54.339417 962 ts_manager.cc:194] Registered new tserver with Master: 9208741a27bd44c2aceabfc382169625 (127.0.98.1:41493)
I20260620 08:59:54.340624 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:59:54.355772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:54.461920 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:54.462193 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:54.462234 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:54.466202 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:54.466274 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:54.466370 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:59:54.475044 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:54.476209 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:54.477317 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:54.486686 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.486693 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.486697 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:54.486889 1158 server_base.cc:1034] running on GCE node
I20260620 08:59:54.487397 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:54.488008 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:54.489187 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781945994489165 us; error 35 us; skew 500 ppm
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:54.491852 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:59:54.492756 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38835/ using document root <none> and password file <none>
I20260620 08:59:54.493321 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:54.493418 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:54.493687 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:54.495814 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "72cf181f80dd4e27aa079b7236d365b3"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "7d9922ac22a0b9602a3feb9e18ce1374"
server_key_iv: "20a7181d7af1dfdfc41fbe8f6bf8f045"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.496379 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72cf181f80dd4e27aa079b7236d365b3"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "7d9922ac22a0b9602a3feb9e18ce1374"
server_key_iv: "20a7181d7af1dfdfc41fbe8f6bf8f045"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.499948 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 08:59:54.502437 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.503566 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:54.503695 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "72cf181f80dd4e27aa079b7236d365b3"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "7d9922ac22a0b9602a3feb9e18ce1374"
server_key_iv: "20a7181d7af1dfdfc41fbe8f6bf8f045"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.503798 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:54.522934 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:54.525949 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:54.526158 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:54.526724 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:54.527511 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:54.527580 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.527649 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:54.527700 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.536648 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42111
I20260620 08:59:54.536666 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42111 every 8 connection(s)
I20260620 08:59:54.537528 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:59:54.541255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1158
I20260620 08:59:54.541385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:59:54.541641 392 external_mini_cluster.cc:1428] Setting key 57b30886088a934a0015c1b432e4395e
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 08:59:54.551456 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.539296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49043 (local address 127.0.98.62:37413)
0620 08:59:54.539462 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.539467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.540329 (+ 862us) server_negotiation.cc:407] Connection header received
0620 08:59:54.541740 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.541745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.541802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.541921 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.543287 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.543814 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.544986 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.545208 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.545317 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.547810 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.547813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.547815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.547838 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.549343 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.549887 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.549895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.549897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.549973 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.550314 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.550319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.550321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.550522 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.550628 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.551139 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.551294 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 08:59:54.552338 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.539715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.2:49043)
0620 08:59:54.540177 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.540219 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.541523 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.542124 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.542136 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.542555 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:54.543143 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.543157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.543936 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.543941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.544860 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.544869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.545353 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.545357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.545524 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.546166 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:54.546182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:54.547635 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:54.549460 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.549466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.549481 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.549771 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.550074 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.550079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.550081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.550218 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.550637 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:54.550646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:54.550889 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.551083 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.551466 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 08:59:54.554741 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260620 08:59:54.555008 1287 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:54.555583 1287 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
I20260620 08:59:54.556794 962 ts_manager.cc:194] Registered new tserver with Master: 72cf181f80dd4e27aa079b7236d365b3 (127.0.98.2:42111)
I20260620 08:59:54.557703 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:59:54.567646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:54.676627 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:54.676884 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:54.676929 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:54.680867 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:54.680939 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:54.681000 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:59:54.686192 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:54.687168 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:54.688107 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:54.699498 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.699486 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:54.699883 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:54.700284 1292 server_base.cc:1034] running on GCE node
I20260620 08:59:54.700831 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:54.701705 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:54.702881 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781945994702852 us; error 40 us; skew 500 ppm
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:54.705724 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:59:54.706589 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43867/ using document root <none> and password file <none>
I20260620 08:59:54.707156 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:54.707214 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:54.707438 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:54.709583 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "51fc4fe66c1748b08d6432afef1a1a66"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "e9255697c96326836d76c2cb6c8e7a9f"
server_key_iv: "8ea9046abd5e1eec58675625e2e17b78"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.710141 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51fc4fe66c1748b08d6432afef1a1a66"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "e9255697c96326836d76c2cb6c8e7a9f"
server_key_iv: "8ea9046abd5e1eec58675625e2e17b78"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.713703 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 08:59:54.715994 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.717245 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:54.717366 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "51fc4fe66c1748b08d6432afef1a1a66"
format_stamp: "Formatted at 2026-06-20 08:59:54 on dist-test-slave-scsf"
server_key: "e9255697c96326836d76c2cb6c8e7a9f"
server_key_iv: "8ea9046abd5e1eec58675625e2e17b78"
server_key_version: "encryptionkey@0"
I20260620 08:59:54.717469 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:54.753908 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:54.756876 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:54.757074 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:54.757676 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:59:54.758493 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:54.758539 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.758616 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:54.758656 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:54.766894 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35689
I20260620 08:59:54.766911 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35689 every 8 connection(s)
I20260620 08:59:54.767743 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:59:54.774492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1292
I20260620 08:59:54.774607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:59:54.774884 392 external_mini_cluster.cc:1428] Setting key c30f7cbde3490ca9475ce8e146a450b5
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:54.780926 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.769520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34411 (local address 127.0.98.62:37413)
0620 08:59:54.769664 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.769668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.770572 (+ 904us) server_negotiation.cc:407] Connection header received
0620 08:59:54.771904 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.771907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.771951 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.772031 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.773219 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.773719 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.775068 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.775330 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.775429 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.777957 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.777960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.777962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.777982 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.779179 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.779636 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.779638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.779639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.779706 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.779977 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.779979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.779980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.780153 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.780225 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.780705 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.780800 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:59:54.781741 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.769905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.3:34411)
0620 08:59:54.770415 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.770466 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.771724 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.772192 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.772199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.772622 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:54.773091 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.773104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.773824 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.773827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.774954 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.774964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.775473 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.775477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.775675 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.776297 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:54.776312 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:54.777820 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:54.779275 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.779279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.779290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.779555 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.779779 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:54.779781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:54.779782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:54.779901 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:54.780235 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:54.780240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:54.780493 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.780665 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.780904 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260620 08:59:54.782759 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260620 08:59:54.783004 1421 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:54.783455 1421 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
I20260620 08:59:54.784439 962 ts_manager.cc:194] Registered new tserver with Master: 51fc4fe66c1748b08d6432afef1a1a66 (127.0.98.3:35689)
I20260620 08:59:54.785050 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34411
I20260620 08:59:54.788574 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:59:54.798795 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.791786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.0.98.62:37413)
0620 08:59:54.791914 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.791919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.792090 (+ 171us) server_negotiation.cc:407] Connection header received
0620 08:59:54.792171 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.792173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.792224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.792314 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.793030 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.793548 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.794372 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.794628 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.794732 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.795935 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.795938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.795941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.795964 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.797140 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.797546 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.797549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.797551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.797626 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.797917 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.797922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.797923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.798143 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.798277 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.798511 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.798631 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:59:54.803236 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:59:54.806223 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:59:54.820334 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.814203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42111 (local address 127.0.0.1:54126)
0620 08:59:54.814529 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.814577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.814699 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.815513 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.815519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.815542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:54.815677 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.815689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817473 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.817476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.818914 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.818923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.819831 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.819835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.819929 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.819953 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.820041 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.820180 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":228,"thread_start_us":86,"threads_started":1}
I20260620 08:59:54.820381 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.815269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.0.98.1:41493)
0620 08:59:54.815627 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.815639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.815655 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:59:54.815709 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.815713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.815902 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.816372 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:54.816556 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817230 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.818791 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.819627 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.819747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.819903 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.820119 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.820218 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":84,"threads_started":1}
I20260620 08:59:54.820577 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.814548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.0.1:42148)
0620 08:59:54.814893 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.814905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.814988 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.815775 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.815778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.815795 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:54.815894 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.815900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817356 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.817362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.819099 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.819106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.820186 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.820190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.820272 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.820299 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.820347 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.820422 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":346,"thread_start_us":101,"threads_started":1}
I20260620 08:59:54.820653 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.814375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.98.2:42111)
0620 08:59:54.814983 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.814995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.815018 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:59:54.815093 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.815098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.815342 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.816113 (+ 771us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:54.816253 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817314 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.819066 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.819686 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.820157 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.820294 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.820344 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.820464 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":92,"threads_started":1}
I20260620 08:59:54.821251 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.814828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.98.3:35689)
0620 08:59:54.815367 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.815378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.815401 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:59:54.815472 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.815477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.815636 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.815839 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:54.816008 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817221 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.819228 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.820021 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.820769 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.820931 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.821001 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.821088 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":108,"threads_started":1}
I20260620 08:59:54.821283 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.814879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41493 (local address 127.0.0.1:57718)
0620 08:59:54.815218 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.815232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.815315 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.816090 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.816094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.816112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:54.816216 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.816225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.817387 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.817391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.818663 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.818674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.819770 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.819774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.819899 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.819946 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.820058 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.821151 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":111,"thread_start_us":76,"threads_started":1}
I20260620 08:59:54.824188 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 642cbd1577604ebca298ea2daeb491d0 (DEFAULT_TABLE table=test-table [id=3c817e1786f24952bbb5002aa260ab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:54.824358 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 642cbd1577604ebca298ea2daeb491d0 (DEFAULT_TABLE table=test-table [id=3c817e1786f24952bbb5002aa260ab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:54.824359 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 642cbd1577604ebca298ea2daeb491d0 (DEFAULT_TABLE table=test-table [id=3c817e1786f24952bbb5002aa260ab8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:54.825273 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642cbd1577604ebca298ea2daeb491d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:54.825361 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642cbd1577604ebca298ea2daeb491d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:54.825361 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642cbd1577604ebca298ea2daeb491d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:54.831403 1438 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: Bootstrap starting.
I20260620 08:59:54.832957 1440 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: Bootstrap starting.
I20260620 08:59:54.834198 1438 tablet_bootstrap.cc:654] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:54.834841 1440 tablet_bootstrap.cc:654] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:54.834901 1438 log.cc:824] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:54.835604 1439 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: Bootstrap starting.
I20260620 08:59:54.835887 1440 log.cc:824] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:54.837509 1439 tablet_bootstrap.cc:654] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:54.837805 1438 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: No bootstrap required, opened a new log
I20260620 08:59:54.837805 1440 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: No bootstrap required, opened a new log
I20260620 08:59:54.837991 1438 ts_tablet_manager.cc:1397] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 08:59:54.838014 1440 ts_tablet_manager.cc:1397] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:59:54.838240 1439 log.cc:824] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:54.839885 1439 tablet_bootstrap.cc:492] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: No bootstrap required, opened a new log
I20260620 08:59:54.840107 1439 ts_tablet_manager.cc:1397] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 08:59:54.842358 1438 raft_consensus.cc:348] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.842358 1440 raft_consensus.cc:348] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.842638 1440 raft_consensus.cc:374] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:54.842638 1438 raft_consensus.cc:374] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:54.842696 1438 raft_consensus.cc:729] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cf181f80dd4e27aa079b7236d365b3, State: Initialized, Role: FOLLOWER
I20260620 08:59:54.842733 1440 raft_consensus.cc:729] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fc4fe66c1748b08d6432afef1a1a66, State: Initialized, Role: FOLLOWER
I20260620 08:59:54.843171 1440 consensus_queue.cc:260] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.843171 1438 consensus_queue.cc:260] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.843348 1439 raft_consensus.cc:348] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.843694 1439 raft_consensus.cc:374] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:54.843791 1439 raft_consensus.cc:729] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9208741a27bd44c2aceabfc382169625, State: Initialized, Role: FOLLOWER
I20260620 08:59:54.844183 1440 ts_tablet_manager.cc:1428] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 08:59:54.844183 1438 ts_tablet_manager.cc:1428] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 08:59:54.844280 1439 consensus_queue.cc:260] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.844621 1287 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
I20260620 08:59:54.844658 1421 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
I20260620 08:59:54.844977 1153 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
I20260620 08:59:54.845427 1439 ts_tablet_manager.cc:1428] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 08:59:54.890486 1446 raft_consensus.cc:482] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:59:54.890676 1446 raft_consensus.cc:504] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.891770 1446 leader_election.cc:288] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72cf181f80dd4e27aa079b7236d365b3 (127.0.98.2:42111), 51fc4fe66c1748b08d6432afef1a1a66 (127.0.98.3:35689)
I20260620 08:59:54.897204 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.892281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50651 (local address 127.0.98.3:35689)
0620 08:59:54.892380 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.892383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.892531 (+ 148us) server_negotiation.cc:407] Connection header received
0620 08:59:54.892696 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.892699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.892747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.892836 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:54.893243 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.894083 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.895777 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.896413 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.896572 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.896635 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.897008 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.897074 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:59:54.897204 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.892134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35881 (local address 127.0.98.2:42111)
0620 08:59:54.892275 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.892279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.892535 (+ 256us) server_negotiation.cc:407] Connection header received
0620 08:59:54.892696 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.892699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.892750 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.892837 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:54.893243 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.894033 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.895777 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.896413 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.896573 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.896636 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.896990 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.897075 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:59:54.897243 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.892039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42111 (local address 127.0.98.1:35881)
0620 08:59:54.892435 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.892467 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.892585 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.892856 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.892859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.892882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:54.893125 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.893132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.894178 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.894182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.895644 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.895655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.896590 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.896594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.896789 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.896843 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.896925 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.897043 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":150,"spinlock_wait_cycles":28288,"thread_start_us":102,"threads_started":1}
I20260620 08:59:54.897243 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.892028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.98.1:50651)
0620 08:59:54.892404 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:54.892421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:54.892585 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:54.892856 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:54.892859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:54.892882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:54.893125 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.893132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.894179 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.894182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.895644 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:54.895655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.896590 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.896594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:54.896784 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.896845 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 08:59:54.896925 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:59:54.897045 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":307,"spinlock_wait_cycles":51072,"thread_start_us":109,"threads_started":1}
I20260620 08:59:54.897919 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642cbd1577604ebca298ea2daeb491d0" candidate_uuid: "9208741a27bd44c2aceabfc382169625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cf181f80dd4e27aa079b7236d365b3" is_pre_election: true
I20260620 08:59:54.898134 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642cbd1577604ebca298ea2daeb491d0" candidate_uuid: "9208741a27bd44c2aceabfc382169625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fc4fe66c1748b08d6432afef1a1a66" is_pre_election: true
I20260620 08:59:54.898245 1241 raft_consensus.cc:2454] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9208741a27bd44c2aceabfc382169625 in term 0.
I20260620 08:59:54.898447 1375 raft_consensus.cc:2454] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9208741a27bd44c2aceabfc382169625 in term 0.
I20260620 08:59:54.898767 1041 leader_election.cc:302] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72cf181f80dd4e27aa079b7236d365b3, 9208741a27bd44c2aceabfc382169625; no voters:
I20260620 08:59:54.899106 1446 raft_consensus.cc:2781] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:59:54.899206 1446 raft_consensus.cc:482] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:59:54.899291 1446 raft_consensus.cc:3037] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:54.900548 1446 raft_consensus.cc:504] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.900844 1446 leader_election.cc:288] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [CANDIDATE]: Term 1 election: Requested vote from peers 72cf181f80dd4e27aa079b7236d365b3 (127.0.98.2:42111), 51fc4fe66c1748b08d6432afef1a1a66 (127.0.98.3:35689)
I20260620 08:59:54.901247 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642cbd1577604ebca298ea2daeb491d0" candidate_uuid: "9208741a27bd44c2aceabfc382169625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fc4fe66c1748b08d6432afef1a1a66"
I20260620 08:59:54.901232 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642cbd1577604ebca298ea2daeb491d0" candidate_uuid: "9208741a27bd44c2aceabfc382169625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cf181f80dd4e27aa079b7236d365b3"
I20260620 08:59:54.901361 1375 raft_consensus.cc:3037] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:54.901362 1241 raft_consensus.cc:3037] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:54.902612 1241 raft_consensus.cc:2454] T 642cbd1577604ebca298ea2daeb491d0 P 72cf181f80dd4e27aa079b7236d365b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9208741a27bd44c2aceabfc382169625 in term 1.
I20260620 08:59:54.902612 1375 raft_consensus.cc:2454] T 642cbd1577604ebca298ea2daeb491d0 P 51fc4fe66c1748b08d6432afef1a1a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9208741a27bd44c2aceabfc382169625 in term 1.
I20260620 08:59:54.902871 1041 leader_election.cc:302] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72cf181f80dd4e27aa079b7236d365b3, 9208741a27bd44c2aceabfc382169625; no voters:
I20260620 08:59:54.903066 1446 raft_consensus.cc:2781] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:54.903306 1446 raft_consensus.cc:686] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [term 1 LEADER]: Becoming Leader. State: Replica: 9208741a27bd44c2aceabfc382169625, State: Running, Role: LEADER
I20260620 08:59:54.903646 1446 consensus_queue.cc:237] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } }
I20260620 08:59:54.906666 962 catalog_manager.cc:5495] T 642cbd1577604ebca298ea2daeb491d0 P 9208741a27bd44c2aceabfc382169625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9208741a27bd44c2aceabfc382169625 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9208741a27bd44c2aceabfc382169625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cf181f80dd4e27aa079b7236d365b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9208741a27bd44c2aceabfc382169625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fc4fe66c1748b08d6432afef1a1a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } health_report { overall_health: UNKNOWN } } }
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:54.928933 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.920631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.1:41493)
0620 08:59:54.920765 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.920771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.920961 (+ 190us) server_negotiation.cc:407] Connection header received
0620 08:59:54.921134 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.921137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.921188 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.921291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.921960 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.922456 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.923277 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.923488 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.923594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.926053 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.926063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.926078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.926099 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.927487 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.927847 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.927852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.927853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.927924 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.928147 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.928149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.928150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.928372 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.928580 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.928726 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.928817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:54.954993 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.946916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.98.1:41493)
0620 08:59:54.947070 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.947075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.947195 (+ 120us) server_negotiation.cc:407] Connection header received
0620 08:59:54.947349 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.947357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.947395 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.947486 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.948132 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.948788 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.949622 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.949837 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.949946 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.952225 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.952228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.952230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.952252 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.953433 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.953968 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.953971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.953972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.954052 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.954283 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.954285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.954286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.954437 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.954553 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.954764 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.954860 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:59:54 dist-test-slave-scsf krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:54.979102 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:54.971538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.98.1:41493)
0620 08:59:54.971691 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:54.971696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:54.971825 (+ 129us) server_negotiation.cc:407] Connection header received
0620 08:59:54.971929 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:54.971931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:54.971971 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:54.972063 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:54.972753 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.973253 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.974019 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:54.974214 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:54.974287 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:54.976624 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:54.976627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:54.976629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:54.976650 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:54.977756 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.978110 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.978113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.978114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.978180 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:54.978387 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:54.978390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:54.978391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:54.978529 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:54.978634 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:54.978792 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 08:59:54.978878 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 08:59:54.980387 1067 tablet_service.cc:2908] Rejecting scan request for tablet 642cbd1577604ebca298ea2daeb491d0: Uninitialized: clean time has not yet been initialized
I20260620 08:59:54.982332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1024
I20260620 08:59:54.988507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1158
I20260620 08:59:54.993932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1292
I20260620 08:59:54.999254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 930
2026-06-20T08:59:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2286 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:55 dist-test-slave-scsf krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:55 dist-test-slave-scsf krb5kdc[1463](info): set up 2 sockets
Jun 20 08:59:55 dist-test-slave-scsf krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260620 08:59:56.016022 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:56.131289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44311
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:56.237138 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:56.237430 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:56.237501 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:56.241444 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:56.241546 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:59:56.241578 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:56.241621 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:56.241679 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:56.241719 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:56.246794 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44311
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:56.247830 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:56.248872 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:56.257645 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.257645 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:56.257850 1476 server_base.cc:1034] running on GCE node
W20260620 08:59:56.257721 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:56.258488 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:56.259575 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:56.260754 1476 hybrid_clock.cc:648] HybridClock initialized: now 1781945996260729 us; error 46 us; skew 500 ppm
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:56.263355 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:56.264218 1476 webserver.cc:460] Webserver started at http://127.0.98.62:39233/ using document root <none> and password file <none>
I20260620 08:59:56.264813 1476 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:56.264899 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:56.265125 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:56.267071 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "90c8fbf8b9194f2994706343d6177258"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d6c039191543d61db4476a026c469107"
server_key_iv: "1f08cc695c07850be2480245d9d62733"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.267637 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "90c8fbf8b9194f2994706343d6177258"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d6c039191543d61db4476a026c469107"
server_key_iv: "1f08cc695c07850be2480245d9d62733"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.271180 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 08:59:56.273511 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.274727 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:59:56.274865 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "90c8fbf8b9194f2994706343d6177258"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d6c039191543d61db4476a026c469107"
server_key_iv: "1f08cc695c07850be2480245d9d62733"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.274972 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:56.303403 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:56.306525 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:56.306730 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:56.314494 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44311
I20260620 08:59:56.314525 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44311 every 8 connection(s)
I20260620 08:59:56.315426 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:56.317353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1476
I20260620 08:59:56.317445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:56.317700 392 external_mini_cluster.cc:1428] Setting key fcea13333f69fc379e6d4028466cbb2d
I20260620 08:59:56.318387 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:56.325529 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Bootstrap starting.
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:56.328131 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:56.329084 1554 log.cc:824] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:56.331387 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: No bootstrap required, opened a new log
I20260620 08:59:56.333513 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.319119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.0.98.62:44311)
0620 08:59:56.319546 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:56.319566 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:56.319608 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:59:56.320498 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:56.320521 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:56.320965 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:56.321342 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:56.322023 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.323580 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.324400 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.325131 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.325298 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.327830 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:56.327836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:56.327851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:56.327882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:56.330360 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.330868 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.330874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.330879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.331044 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.331305 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.331308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.331310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.331800 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:56.331983 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:56.332243 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 08:59:56.332563 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260620 08:59:56.335218 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } }
I20260620 08:59:56.335438 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:56.335541 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c8fbf8b9194f2994706343d6177258, State: Initialized, Role: FOLLOWER
I20260620 08:59:56.336018 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } }
I20260620 08:59:56.336155 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:56.336239 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:56.336334 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:56.337402 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } }
I20260620 08:59:56.337749 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c8fbf8b9194f2994706343d6177258; no voters:
I20260620 08:59:56.338069 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:56.338186 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:56.338424 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [term 1 LEADER]: Becoming Leader. State: Replica: 90c8fbf8b9194f2994706343d6177258, State: Running, Role: LEADER
I20260620 08:59:56.338784 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } }
I20260620 08:59:56.339272 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:56.340395 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c8fbf8b9194f2994706343d6177258. Latest consensus state: current_term: 1 leader_uuid: "90c8fbf8b9194f2994706343d6177258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } } }
I20260620 08:59:56.340535 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:56.340801 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90c8fbf8b9194f2994706343d6177258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c8fbf8b9194f2994706343d6177258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44311 } } }
I20260620 08:59:56.340883 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258 [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:56.340888 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:56.343410 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:56.349252 1568 catalog_manager.cc:1347] Generated new cluster ID: 58660ac48fca44eea48de4e3254ba56f
I20260620 08:59:56.349323 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:56.361713 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:56.362946 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:56.372287 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Generated new TSK 0
I20260620 08:59:56.373018 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:59:56.407534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44311
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:56.518365 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:56.518656 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:56.518702 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:56.522919 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:56.522987 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:56.523054 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:59:56.523126 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:59:56.528416 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:56.529601 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:56.530666 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:56.539949 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.539949 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.539950 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:56.540221 1580 server_base.cc:1034] running on GCE node
I20260620 08:59:56.540756 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:56.541353 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:56.542510 1580 hybrid_clock.cc:648] HybridClock initialized: now 1781945996542505 us; error 53 us; skew 500 ppm
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:56.545184 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:59:56.546034 1580 webserver.cc:460] Webserver started at http://127.0.98.1:40459/ using document root <none> and password file <none>
I20260620 08:59:56.546594 1580 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:56.546685 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:56.546909 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:56.548956 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "33870f19f96e4a77b8d9a20a3ece753d"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "1787f228b44a142f866bd576ceee8c33"
server_key_iv: "462e14c3cf67ee4b599f7045f09d2664"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.549523 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33870f19f96e4a77b8d9a20a3ece753d"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "1787f228b44a142f866bd576ceee8c33"
server_key_iv: "462e14c3cf67ee4b599f7045f09d2664"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.553223 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 08:59:56.556010 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.557360 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:59:56.557483 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "33870f19f96e4a77b8d9a20a3ece753d"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "1787f228b44a142f866bd576ceee8c33"
server_key_iv: "462e14c3cf67ee4b599f7045f09d2664"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.557606 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:56.577363 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:56.580555 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:56.580752 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:56.582155 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:56.582230 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.582297 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:56.582348 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:56.592990 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34411
I20260620 08:59:56.593044 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34411 every 8 connection(s)
I20260620 08:59:56.594050 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:59:56.596992 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.583020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50991 (local address 127.0.98.62:44311)
0620 08:59:56.583195 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:56.583202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:56.584040 (+ 838us) server_negotiation.cc:407] Connection header received
0620 08:59:56.585217 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:56.585221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:56.585274 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:56.585322 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:56.587106 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.587941 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.589838 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.590093 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.590183 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.593300 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:56.593302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:56.593305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:56.593328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:56.594904 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.595420 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.595423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.595425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.595513 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.595865 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.595868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.595870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.596058 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:56.596162 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:56.596696 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 08:59:56.596823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 08:59:56.597749 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.583407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.1:50991)
0620 08:59:56.583876 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:56.583926 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:56.584977 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:56.585740 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:56.585749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:56.586259 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:56.586964 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.586984 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.588091 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.588095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.589701 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.589715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.590263 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.590269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.590502 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.591201 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:56.591224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:56.593091 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:56.595028 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.595049 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.595064 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.595309 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.595611 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.595615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.595617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.595761 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.596170 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:56.596179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:56.596410 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 08:59:56.596643 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 08:59:56.596949 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260620 08:59:56.598927 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44311
I20260620 08:59:56.599099 1711 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:56.599588 1711 heartbeater.cc:511] Master 127.0.98.62:44311 requested a full tablet report, sending...
I20260620 08:59:56.601181 1507 ts_manager.cc:194] Registered new tserver with Master: 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411)
I20260620 08:59:56.602505 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50991
I20260620 08:59:56.604131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1580
I20260620 08:59:56.604264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:59:56.604611 392 external_mini_cluster.cc:1428] Setting key 3dadd8029e603e05ac41ff5ce4c4a619
I20260620 08:59:56.608114 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.601021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.1:43781)
0620 08:59:56.601321 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:56.601335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:56.601432 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:56.601665 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:56.601668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:56.601862 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:56.601977 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.601984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.602696 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.602699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.603340 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.603349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.603781 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.603784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.603895 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.604471 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:56.604488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:56.604915 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:56.606612 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.606618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.606623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.606889 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.607156 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.607162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.607167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.607267 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.607702 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:56.607704 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:56.607778 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 08:59:56.607861 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:59:56.607965 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260620 08:59:56.608264 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.601135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43781 (local address 127.0.98.62:44311)
0620 08:59:56.601249 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:56.601253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:56.601394 (+ 141us) server_negotiation.cc:407] Connection header received
0620 08:59:56.601522 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:56.601524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:56.601562 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:56.601624 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:56.602090 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.602572 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.603451 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.603670 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.603778 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.605056 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:56.605059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:56.605061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:56.605080 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:56.606498 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.607000 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.607002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.607003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.607073 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.607358 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.607363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.607366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.607604 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:56.607708 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:56.607941 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:59:56.608074 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 08:59:56.629314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44311
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:56.739871 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:56.740209 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:56.740288 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:56.744566 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:56.744616 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:56.744654 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:59:56.744693 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 08:59:56.749861 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:56.750847 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:56.751811 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:56.762130 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.762095 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.762095 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:56.762620 1720 server_base.cc:1034] running on GCE node
I20260620 08:59:56.763053 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:56.763650 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:56.764846 1720 hybrid_clock.cc:648] HybridClock initialized: now 1781945996764831 us; error 32 us; skew 500 ppm
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:56.767457 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 08:59:56.768242 1720 webserver.cc:460] Webserver started at http://127.0.98.2:43129/ using document root <none> and password file <none>
I20260620 08:59:56.768824 1720 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:56.768885 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:56.769049 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:56.770974 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "14817ea8f78646cf99b2c6bfe0da5a06"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "8dac9a642cace71be3418209bbe9f23a"
server_key_iv: "b257f92ed8ad042deca0df1af35e90bc"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.771477 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14817ea8f78646cf99b2c6bfe0da5a06"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "8dac9a642cace71be3418209bbe9f23a"
server_key_iv: "b257f92ed8ad042deca0df1af35e90bc"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.775132 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:59:56.777412 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.778642 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:59:56.778766 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "14817ea8f78646cf99b2c6bfe0da5a06"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "8dac9a642cace71be3418209bbe9f23a"
server_key_iv: "b257f92ed8ad042deca0df1af35e90bc"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.778862 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:56.798979 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:56.802218 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:56.802424 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:56.803896 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:56.803974 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.804057 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:56.804107 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:56.814826 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45559
I20260620 08:59:56.815469 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45559 every 8 connection(s)
I20260620 08:59:56.816009 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 08:59:56.818320 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.804917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56743 (local address 127.0.98.62:44311)
0620 08:59:56.805089 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:56.805094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:56.806064 (+ 970us) server_negotiation.cc:407] Connection header received
0620 08:59:56.807472 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:56.807477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:56.807543 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:56.807643 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:56.809378 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.809977 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.811133 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.811352 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.811479 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.814441 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:56.814444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:56.814446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:56.814469 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:56.816344 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.816921 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.816926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.816927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.817021 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.817379 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.817382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.817383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.817577 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:56.817657 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:56.818102 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 08:59:56.818208 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 08:59:56.819101 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.805382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.2:56743)
0620 08:59:56.805881 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:56.805941 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:56.807210 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:56.807867 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:56.807878 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:56.808406 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:56.809229 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.809253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.810120 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.810124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.811005 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.811014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.811537 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.811542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.811711 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.812838 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:56.812854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:56.814253 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:56.816476 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.816488 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.816504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.816835 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.817126 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.817129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.817131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.817272 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.817668 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:56.817672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:56.817881 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 08:59:56.818042 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 08:59:56.818288 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260620 08:59:56.820328 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44311
I20260620 08:59:56.820503 1851 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:56.820994 1851 heartbeater.cc:511] Master 127.0.98.62:44311 requested a full tablet report, sending...
I20260620 08:59:56.822032 1508 ts_manager.cc:194] Registered new tserver with Master: 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2:45559)
I20260620 08:59:56.822711 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56743
I20260620 08:59:56.825330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1720
I20260620 08:59:56.825457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 08:59:56.825790 392 external_mini_cluster.cc:1428] Setting key a786b04e0686cd31c96ba82391c3d810
I20260620 08:59:56.829766 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.822943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32997 (local address 127.0.98.62:44311)
0620 08:59:56.823068 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:56.823071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:56.823081 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:59:56.823108 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:56.823110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:56.823139 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:56.823232 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:56.823643 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.824127 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.824996 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.825208 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.825311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.826504 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:56.826506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:56.826508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:56.826529 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:56.828154 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.828593 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.828599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.828604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.828691 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:56.829006 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:56.829008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:56.829009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:56.829195 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:56.829270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:56.829580 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 08:59:56.829665 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:59:56.829914 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:56.822360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.2:32997)
0620 08:59:56.822634 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:56.822646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:56.822714 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:56.823247 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:56.823250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:56.823438 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:56.823532 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.823538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.824247 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.824252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.824859 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:56.824867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:56.825315 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:56.825318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:56.825409 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:56.825966 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:56.825977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:56.826379 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:56.828260 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.828262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.828264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.828442 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.828793 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:56.828799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:56.828802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:56.828905 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:56.829286 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:56.829291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:56.829392 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 08:59:56.829531 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 08:59:56.829685 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 08:59:56.851400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44311
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:56.961862 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:56.962178 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:56.962229 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:56.966178 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:56.966228 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:56.966265 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:59:56.966297 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 08:59:56.971427 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:56.972383 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:56.973387 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:56.982689 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.982733 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:56.982728 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:56.983022 1860 server_base.cc:1034] running on GCE node
I20260620 08:59:56.983350 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:56.983968 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:56.985129 1860 hybrid_clock.cc:648] HybridClock initialized: now 1781945996985097 us; error 47 us; skew 500 ppm
Jun 20 08:59:56 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:56.987682 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 08:59:56.988590 1860 webserver.cc:460] Webserver started at http://127.0.98.3:41471/ using document root <none> and password file <none>
I20260620 08:59:56.989146 1860 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:56.989233 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:56.989451 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:56.991436 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7ad4ae809f046e8bb319c2935caafb6"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d0b7a56304cd63c951fdaa2b55af9d5b"
server_key_iv: "5fcdd63a3d6fdaf260b07ea683479b3b"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.992003 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7ad4ae809f046e8bb319c2935caafb6"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d0b7a56304cd63c951fdaa2b55af9d5b"
server_key_iv: "5fcdd63a3d6fdaf260b07ea683479b3b"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.995540 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 08:59:56.997927 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:56.999161 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:59:56.999286 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "c7ad4ae809f046e8bb319c2935caafb6"
format_stamp: "Formatted at 2026-06-20 08:59:56 on dist-test-slave-scsf"
server_key: "d0b7a56304cd63c951fdaa2b55af9d5b"
server_key_iv: "5fcdd63a3d6fdaf260b07ea683479b3b"
server_key_version: "encryptionkey@0"
I20260620 08:59:56.999392 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:57.009845 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:57.012665 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:57.012856 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:57.014166 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:57.014240 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:57.014307 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:57.014355 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:59:57 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:57.025069 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36223
I20260620 08:59:57.025136 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36223 every 8 connection(s)
I20260620 08:59:57.026109 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 08:59:57.027382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 1860
I20260620 08:59:57.027539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 08:59:57.027802 392 external_mini_cluster.cc:1428] Setting key fa9d8f492ee749e37bd780017f85b771
I20260620 08:59:57.028649 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.015118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51267 (local address 127.0.98.62:44311)
0620 08:59:57.015242 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.015246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.016252 (+ 1006us) server_negotiation.cc:407] Connection header received
0620 08:59:57.017508 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.017512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.017575 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.017666 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:57.019202 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.020053 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.021561 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.021864 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.021995 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.024773 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:57.024778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:57.024785 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:57.024813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:57.026362 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.026921 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.026927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.026929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.027028 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.027368 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.027371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.027373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.027635 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:57.027731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.028312 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.028474 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:59:57.029421 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.015624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.3:51267)
0620 08:59:57.016098 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.016137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.017233 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.017860 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.017871 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.018319 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:57.019018 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.019033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.020172 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.020194 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.021414 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.021423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.022042 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.022046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.022241 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.022857 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:57.022873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:57.024609 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:57.026485 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.026491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.026504 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.026830 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.027123 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.027126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.027127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.027252 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.027727 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:57.027733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:57.028013 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.028210 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.028547 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20260620 08:59:57.030504 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44311
I20260620 08:59:57.030632 1991 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:57.031147 1991 heartbeater.cc:511] Master 127.0.98.62:44311 requested a full tablet report, sending...
I20260620 08:59:57.032176 1508 ts_manager.cc:194] Registered new tserver with Master: c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:57.032956 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51267
I20260620 08:59:57.039389 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.032686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.98.3:57609)
0620 08:59:57.032982 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.032998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.033079 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.033426 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.033429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.033625 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:57.033714 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.033719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.034438 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.034441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.034992 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.034999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.035458 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.035462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.035554 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.035997 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:57.036008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:57.036303 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:57.037852 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.037858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.037861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.038062 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.038340 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.038344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.038346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.038428 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.038865 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:57.038868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:57.038961 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.039065 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.039202 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260620 08:59:57.039383 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.032802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57609 (local address 127.0.98.62:44311)
0620 08:59:57.033255 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.033258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.033267 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:59:57.033292 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.033294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.033323 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.033410 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:57.033825 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.034319 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.035113 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.035341 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.035453 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.036427 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:57.036431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:57.036433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:57.036486 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:57.037741 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.038161 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.038166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.038167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.038248 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.038524 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.038527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.038529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.038695 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:57.038848 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.039128 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.039230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260620 08:59:57.041678 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:59:57 dist-test-slave-scsf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:59:57 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:57.075814 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.067476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.0.98.62:44311)
0620 08:59:57.067601 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.067606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.067627 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:59:57.067670 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.067672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.067713 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.067814 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:57.068876 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.069541 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.070372 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.070581 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.070662 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.073063 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:57.073066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:57.073068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:57.073090 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:57.074258 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.074626 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.074628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.074629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.074698 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.074914 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.074917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.074918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.075140 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:57.075244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.075492 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.075580 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:59:57.078734 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:59:57.082091 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:59:57.096312 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.0.98.2:45559)
0620 08:59:57.091380 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.091392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.091415 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:59:57.091494 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.091499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.091669 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.091821 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:57.092096 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093363 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.094939 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095547 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.095694 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.095863 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.096061 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.096152 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":106,"threads_started":1}
I20260620 08:59:57.096311 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34411 (local address 127.0.0.1:38116)
0620 08:59:57.091033 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.091051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.091143 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.091912 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.091915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.091932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:57.092031 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.092038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093551 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.093553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.094634 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.094643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095723 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.095726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.095855 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.095905 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.095988 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.096143 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":133,"thread_start_us":96,"threads_started":1}
I20260620 08:59:57.096582 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45559 (local address 127.0.0.1:38372)
0620 08:59:57.090539 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.090587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.090711 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.091820 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.091826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.091846 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:57.091973 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.091980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093538 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.093544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.094804 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.094811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095712 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.095716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.095818 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.095903 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.096030 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.096137 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":100,"thread_start_us":105,"threads_started":1}
I20260620 08:59:57.096326 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36223 (local address 127.0.0.1:38312)
0620 08:59:57.091314 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.091327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.091414 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.091978 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.091980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.092036 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:57.092131 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.092135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093138 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.093143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.094695 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.094705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095950 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.095954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.096036 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.096055 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.096119 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.096177 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":54,"thread_start_us":81,"threads_started":1}
I20260620 08:59:57.096773 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.3:36223)
0620 08:59:57.091439 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.091452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.091477 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:59:57.091558 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.091563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.091754 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.091906 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:57.092236 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093024 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.094808 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095580 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.095699 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.095892 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.096391 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.096510 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260620 08:59:57.096918 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.090829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.0.98.1:34411)
0620 08:59:57.091478 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.091491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.091511 (+ 20us) server_negotiation.cc:407] Connection header received
0620 08:59:57.091580 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.091584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.091753 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.092185 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:57.092305 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.093404 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.094757 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.095579 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.096226 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.096381 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.096425 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.096538 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":77,"threads_started":1}
I20260620 08:59:57.099512 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 2581020cde504c5987cec06c3bc44968 (DEFAULT_TABLE table=test-table [id=fc9874eaf23346ebb6d42555180f00c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.099548 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 2581020cde504c5987cec06c3bc44968 (DEFAULT_TABLE table=test-table [id=fc9874eaf23346ebb6d42555180f00c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.099557 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 2581020cde504c5987cec06c3bc44968 (DEFAULT_TABLE table=test-table [id=fc9874eaf23346ebb6d42555180f00c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.100562 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2581020cde504c5987cec06c3bc44968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.100561 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2581020cde504c5987cec06c3bc44968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.100625 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2581020cde504c5987cec06c3bc44968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.106827 2014 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Bootstrap starting.
I20260620 08:59:57.108174 2015 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Bootstrap starting.
I20260620 08:59:57.109556 2016 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Bootstrap starting.
I20260620 08:59:57.109766 2014 tablet_bootstrap.cc:654] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.110323 2015 tablet_bootstrap.cc:654] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.110646 2014 log.cc:824] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:57.111115 2015 log.cc:824] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:57.111521 2016 tablet_bootstrap.cc:654] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.112264 2016 log.cc:824] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:57.112751 2014 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: No bootstrap required, opened a new log
I20260620 08:59:57.112969 2015 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: No bootstrap required, opened a new log
I20260620 08:59:57.112984 2014 ts_tablet_manager.cc:1397] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 08:59:57.113186 2015 ts_tablet_manager.cc:1397] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:59:57.114138 2016 tablet_bootstrap.cc:492] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: No bootstrap required, opened a new log
I20260620 08:59:57.114379 2016 ts_tablet_manager.cc:1397] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:59:57.116299 2015 raft_consensus.cc:348] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.116595 2015 raft_consensus.cc:374] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.116670 2015 raft_consensus.cc:729] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ad4ae809f046e8bb319c2935caafb6, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.116801 2014 raft_consensus.cc:348] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.117058 2014 raft_consensus.cc:374] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.117118 2014 raft_consensus.cc:729] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33870f19f96e4a77b8d9a20a3ece753d, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.117167 2015 consensus_queue.cc:260] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.117560 2014 consensus_queue.cc:260] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.117790 1991 heartbeater.cc:503] Master 127.0.98.62:44311 was elected leader, sending a full tablet report...
I20260620 08:59:57.118176 2015 ts_tablet_manager.cc:1428] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:59:57.118497 2014 ts_tablet_manager.cc:1428] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:59:57.118479 2016 raft_consensus.cc:348] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.118669 1711 heartbeater.cc:503] Master 127.0.98.62:44311 was elected leader, sending a full tablet report...
I20260620 08:59:57.118727 2016 raft_consensus.cc:374] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.118845 2016 raft_consensus.cc:729] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14817ea8f78646cf99b2c6bfe0da5a06, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.119241 2016 consensus_queue.cc:260] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.119966 1851 heartbeater.cc:503] Master 127.0.98.62:44311 was elected leader, sending a full tablet report...
I20260620 08:59:57.121289 2016 ts_tablet_manager.cc:1428] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260620 08:59:57.277141 1992 tablet.cc:2369] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:59:57.317668 1852 tablet.cc:2369] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:59:57.345500 1712 tablet.cc:2369] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:59:57.396752 2022 raft_consensus.cc:482] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:59:57.397018 2022 raft_consensus.cc:504] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.398118 2022 leader_election.cc:288] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:57.403569 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.398558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43949 (local address 127.0.98.1:34411)
0620 08:59:57.398685 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.398689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.398867 (+ 178us) server_negotiation.cc:407] Connection header received
0620 08:59:57.399026 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.399028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.399075 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.399169 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:57.399532 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.400318 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402087 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.402698 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402802 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.402866 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.403377 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.403444 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:59:57.403569 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.398568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60479 (local address 127.0.98.3:36223)
0620 08:59:57.398685 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.398689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.398867 (+ 178us) server_negotiation.cc:407] Connection header received
0620 08:59:57.399018 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.399021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.399075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.399168 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:57.399532 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.400315 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402087 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.402704 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402802 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.402866 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.403377 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.403444 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:59:57.403607 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.398388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34411 (local address 127.0.98.2:43949)
0620 08:59:57.398776 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.398792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.398900 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.399194 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.399197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.399223 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:57.399408 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.399415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.400435 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.400438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.401960 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.401972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.402928 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.403167 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.403263 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.403342 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.403441 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":299,"spinlock_wait_cycles":26240,"thread_start_us":65,"threads_started":1}
I20260620 08:59:57.403650 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.398388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36223 (local address 127.0.98.2:60479)
0620 08:59:57.398747 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.398763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.398900 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.399194 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.399197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.399222 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:57.399420 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.399425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.400435 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.400438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.401960 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.401972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.402928 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.402932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.403166 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.403263 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.403342 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.403426 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":228,"spinlock_wait_cycles":15744,"thread_start_us":74,"threads_started":1}
I20260620 08:59:57.404547 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2581020cde504c5987cec06c3bc44968" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33870f19f96e4a77b8d9a20a3ece753d" is_pre_election: true
I20260620 08:59:57.404546 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2581020cde504c5987cec06c3bc44968" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6" is_pre_election: true
I20260620 08:59:57.404884 1665 raft_consensus.cc:2454] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 0.
I20260620 08:59:57.404884 1945 raft_consensus.cc:2454] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 0.
I20260620 08:59:57.405428 1736 leader_election.cc:302] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14817ea8f78646cf99b2c6bfe0da5a06, 33870f19f96e4a77b8d9a20a3ece753d, c7ad4ae809f046e8bb319c2935caafb6; no voters:
I20260620 08:59:57.405732 2022 raft_consensus.cc:2781] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:59:57.405836 2022 raft_consensus.cc:482] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:59:57.405896 2022 raft_consensus.cc:3037] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.406956 2022 raft_consensus.cc:504] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.407323 2022 leader_election.cc:288] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 election: Requested vote from peers 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:57.407729 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2581020cde504c5987cec06c3bc44968" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33870f19f96e4a77b8d9a20a3ece753d"
I20260620 08:59:57.407722 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2581020cde504c5987cec06c3bc44968" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6"
I20260620 08:59:57.407852 1665 raft_consensus.cc:3037] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.407852 1945 raft_consensus.cc:3037] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.409235 1665 raft_consensus.cc:2454] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 1.
I20260620 08:59:57.409569 1945 raft_consensus.cc:2454] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 1.
I20260620 08:59:57.409610 1737 leader_election.cc:302] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14817ea8f78646cf99b2c6bfe0da5a06, 33870f19f96e4a77b8d9a20a3ece753d; no voters:
I20260620 08:59:57.409816 2022 raft_consensus.cc:2781] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:57.410017 2022 raft_consensus.cc:686] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 LEADER]: Becoming Leader. State: Replica: 14817ea8f78646cf99b2c6bfe0da5a06, State: Running, Role: LEADER
I20260620 08:59:57.410368 2022 consensus_queue.cc:237] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.413499 1507 catalog_manager.cc:5495] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260620 08:59:57.469990 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.466653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.98.2:45559)
0620 08:59:57.466778 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.466781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.466794 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:59:57.466960 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.466963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.467020 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.467115 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:57.467344 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.467836 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.468719 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.468917 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.468991 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.469263 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:57.469684 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:57.469769 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.469834 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.469881 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:59:57.475859 1945 raft_consensus.cc:1261] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Refusing update from remote peer 14817ea8f78646cf99b2c6bfe0da5a06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:57.475859 1665 raft_consensus.cc:1261] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Refusing update from remote peer 14817ea8f78646cf99b2c6bfe0da5a06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:57.476655 2025 consensus_queue.cc:1035] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:57.476861 2022 consensus_queue.cc:1035] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:57.489161 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298850805652799488 to 7298850805395587072. Current Snapshot: MvccSnapshot[applied={T|T < 7298850805652799488 or (T in {7298850805652799488})}]
I20260620 08:59:57.489372 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298850805652799488 to 7298850805395587072. Current Snapshot: MvccSnapshot[applied={T|T < 7298850805652799488 or (T in {7298850805652799488})}]
I20260620 08:59:57.496165 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:59:57.496308 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:59:57.498615 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Sending DeleteTablet for 3 replicas of tablet 2581020cde504c5987cec06c3bc44968
I20260620 08:59:57.499348 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 2581020cde504c5987cec06c3bc44968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 08:59:57.499373 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 2581020cde504c5987cec06c3bc44968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38116
I20260620 08:59:57.499516 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 2581020cde504c5987cec06c3bc44968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38312
I20260620 08:59:57.500000 2039 tablet_replica.cc:330] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: stopping tablet replica
I20260620 08:59:57.500088 2037 tablet_replica.cc:330] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: stopping tablet replica
I20260620 08:59:57.500375 2039 raft_consensus.cc:2229] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:59:57.500370 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:59:57.500738 2039 raft_consensus.cc:2258] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 08:59:57.500828 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:59:57.501287 2038 tablet_replica.cc:330] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: stopping tablet replica
I20260620 08:59:57.501578 2038 raft_consensus.cc:2229] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:59:57.501852 2038 raft_consensus.cc:2258] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:59:57.502338 2037 raft_consensus.cc:2229] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:59:57.502708 2037 raft_consensus.cc:2258] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:59:57.503304 2039 ts_tablet_manager.cc:1905] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:57.503899 2038 ts_tablet_manager.cc:1905] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:57.506163 2039 ts_tablet_manager.cc:1918] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:59:57.506290 2039 log.cc:1196] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/2581020cde504c5987cec06c3bc44968
I20260620 08:59:57.506613 2038 ts_tablet_manager.cc:1918] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:59:57.506646 2039 ts_tablet_manager.cc:1939] T 2581020cde504c5987cec06c3bc44968 P c7ad4ae809f046e8bb319c2935caafb6: Deleting consensus metadata
I20260620 08:59:57.506742 2038 log.cc:1196] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/2581020cde504c5987cec06c3bc44968
I20260620 08:59:57.507053 2038 ts_tablet_manager.cc:1939] T 2581020cde504c5987cec06c3bc44968 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting consensus metadata
I20260620 08:59:57.507134 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 98c364ff8111471192ab641fa10fad07 (DEFAULT_TABLE table=test-table [id=b45200273118468e8af061cc6c683975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.507253 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 98c364ff8111471192ab641fa10fad07 (DEFAULT_TABLE table=test-table [id=b45200273118468e8af061cc6c683975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.507398 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c364ff8111471192ab641fa10fad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.507498 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c364ff8111471192ab641fa10fad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.508162 1493 catalog_manager.cc:4841] TS 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411): tablet 2581020cde504c5987cec06c3bc44968 (table test-table [id=fc9874eaf23346ebb6d42555180f00c5]) successfully deleted
I20260620 08:59:57.509524 2014 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Bootstrap starting.
I20260620 08:59:57.509905 2016 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Bootstrap starting.
I20260620 08:59:57.509966 1492 catalog_manager.cc:4841] TS c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223): tablet 2581020cde504c5987cec06c3bc44968 (table test-table [id=fc9874eaf23346ebb6d42555180f00c5]) successfully deleted
I20260620 08:59:57.510422 2014 tablet_bootstrap.cc:654] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.510448 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 98c364ff8111471192ab641fa10fad07 (DEFAULT_TABLE table=test-table [id=b45200273118468e8af061cc6c683975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:59:57.510772 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c364ff8111471192ab641fa10fad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.510974 2016 tablet_bootstrap.cc:654] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.512274 2016 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: No bootstrap required, opened a new log
I20260620 08:59:57.512362 2016 ts_tablet_manager.cc:1397] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:57.512857 2014 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: No bootstrap required, opened a new log
I20260620 08:59:57.512806 2016 raft_consensus.cc:348] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.512909 2016 raft_consensus.cc:374] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.512938 2016 raft_consensus.cc:729] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14817ea8f78646cf99b2c6bfe0da5a06, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.512976 2014 ts_tablet_manager.cc:1397] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:59:57.513073 2016 consensus_queue.cc:260] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.513139 2015 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Bootstrap starting.
I20260620 08:59:57.513301 2016 ts_tablet_manager.cc:1428] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:59:57.513547 2014 raft_consensus.cc:348] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.513656 2014 raft_consensus.cc:374] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.513693 2014 raft_consensus.cc:729] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33870f19f96e4a77b8d9a20a3ece753d, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.513833 2014 consensus_queue.cc:260] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.514099 2015 tablet_bootstrap.cc:654] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.514132 2014 ts_tablet_manager.cc:1428] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:59:57.516812 2037 ts_tablet_manager.cc:1905] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:57.518901 2037 ts_tablet_manager.cc:1918] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:59:57.519011 2037 log.cc:1196] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/2581020cde504c5987cec06c3bc44968
I20260620 08:59:57.519300 2037 ts_tablet_manager.cc:1939] T 2581020cde504c5987cec06c3bc44968 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting consensus metadata
I20260620 08:59:57.520615 2015 tablet_bootstrap.cc:492] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: No bootstrap required, opened a new log
I20260620 08:59:57.520709 2015 ts_tablet_manager.cc:1397] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 08:59:57.521020 1495 catalog_manager.cc:4841] TS 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2:45559): tablet 2581020cde504c5987cec06c3bc44968 (table test-table [id=fc9874eaf23346ebb6d42555180f00c5]) successfully deleted
I20260620 08:59:57.521272 2015 raft_consensus.cc:348] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.521390 2015 raft_consensus.cc:374] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.521432 2015 raft_consensus.cc:729] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ad4ae809f046e8bb319c2935caafb6, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.521566 2015 consensus_queue.cc:260] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.521842 2015 ts_tablet_manager.cc:1428] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:59:57.721220 2025 raft_consensus.cc:482] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:59:57.721369 2025 raft_consensus.cc:504] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.721884 2025 leader_election.cc:288] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:57.722378 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c364ff8111471192ab641fa10fad07" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6" is_pre_election: true
I20260620 08:59:57.722388 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c364ff8111471192ab641fa10fad07" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33870f19f96e4a77b8d9a20a3ece753d" is_pre_election: true
I20260620 08:59:57.722512 1945 raft_consensus.cc:2454] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 0.
I20260620 08:59:57.722573 1665 raft_consensus.cc:2454] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 0.
I20260620 08:59:57.722877 1736 leader_election.cc:302] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14817ea8f78646cf99b2c6bfe0da5a06, c7ad4ae809f046e8bb319c2935caafb6; no voters:
I20260620 08:59:57.723062 2025 raft_consensus.cc:2781] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:59:57.723125 2025 raft_consensus.cc:482] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:59:57.723169 2025 raft_consensus.cc:3037] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.724125 2025 raft_consensus.cc:504] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.724443 2025 leader_election.cc:288] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 election: Requested vote from peers 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:57.724838 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c364ff8111471192ab641fa10fad07" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33870f19f96e4a77b8d9a20a3ece753d"
I20260620 08:59:57.724877 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c364ff8111471192ab641fa10fad07" candidate_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6"
I20260620 08:59:57.724934 1665 raft_consensus.cc:3037] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.725028 1945 raft_consensus.cc:3037] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:57.725831 1665 raft_consensus.cc:2454] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 1.
I20260620 08:59:57.726018 1945 raft_consensus.cc:2454] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14817ea8f78646cf99b2c6bfe0da5a06 in term 1.
I20260620 08:59:57.726152 1737 leader_election.cc:302] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14817ea8f78646cf99b2c6bfe0da5a06, 33870f19f96e4a77b8d9a20a3ece753d; no voters:
I20260620 08:59:57.726331 2025 raft_consensus.cc:2781] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:57.726445 2025 raft_consensus.cc:686] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 LEADER]: Becoming Leader. State: Replica: 14817ea8f78646cf99b2c6bfe0da5a06, State: Running, Role: LEADER
I20260620 08:59:57.726586 2025 consensus_queue.cc:237] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.728236 1508 catalog_manager.cc:5495] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } health_report { overall_health: UNKNOWN } } }
Jun 20 08:59:57 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:57.782982 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.774143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.98.62:44311)
0620 08:59:57.774478 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:57.774487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:57.774590 (+ 103us) server_negotiation.cc:407] Connection header received
0620 08:59:57.774745 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:57.774748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:57.774793 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:57.774885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:57.775519 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.776051 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.776957 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.777186 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.777293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.779616 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:57.779619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:57.779622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:57.779644 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:57.781093 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.781580 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.781586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.781588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.781688 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:57.781957 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:57.781962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:57.781964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:57.782217 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:57.782317 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:57.782627 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 08:59:57.782821 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260620 08:59:57.782982 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:57.774133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44311 (local address 127.0.0.1:55648)
0620 08:59:57.774498 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:57.774514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:57.774647 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:57.774899 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:57.774902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:57.775317 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:57.775404 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.775410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.776156 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.776162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.776846 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:57.776855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:57.777297 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:57.777300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:57.777383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:57.778015 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:57.778027 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:57.779471 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:57.781202 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.781213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.781220 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.781454 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.781784 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:57.781787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:57.781789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:57.781868 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:57.782323 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:57.782332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:57.782440 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 08:59:57.782574 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 08:59:57.782758 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"spinlock_wait_cycles":13056,"thread_start_us":127,"threads_started":1}
I20260620 08:59:57.786832 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55648:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:59:57.787563 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:59:57.792194 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 08deb28cb0dd4eabaff00c04104eae3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc4067a3a424488bbe575c7727e9e1c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:59:57.792255 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 08deb28cb0dd4eabaff00c04104eae3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc4067a3a424488bbe575c7727e9e1c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:59:57.792321 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 08deb28cb0dd4eabaff00c04104eae3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc4067a3a424488bbe575c7727e9e1c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:59:57.792680 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08deb28cb0dd4eabaff00c04104eae3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.792680 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08deb28cb0dd4eabaff00c04104eae3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.792683 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08deb28cb0dd4eabaff00c04104eae3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:57.795297 2014 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: Bootstrap starting.
I20260620 08:59:57.795994 2015 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: Bootstrap starting.
I20260620 08:59:57.796331 2014 tablet_bootstrap.cc:654] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.796435 2016 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: Bootstrap starting.
I20260620 08:59:57.797246 2016 tablet_bootstrap.cc:654] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.797384 2015 tablet_bootstrap.cc:654] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:57.798547 2016 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: No bootstrap required, opened a new log
I20260620 08:59:57.798564 2014 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: No bootstrap required, opened a new log
I20260620 08:59:57.798666 2014 ts_tablet_manager.cc:1397] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:59:57.798666 2016 ts_tablet_manager.cc:1397] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:59:57.798867 2015 tablet_bootstrap.cc:492] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: No bootstrap required, opened a new log
I20260620 08:59:57.798952 2015 ts_tablet_manager.cc:1397] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 08:59:57.799116 2016 raft_consensus.cc:348] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799141 2014 raft_consensus.cc:348] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799254 2016 raft_consensus.cc:374] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.799261 2014 raft_consensus.cc:374] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.799296 2014 raft_consensus.cc:729] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33870f19f96e4a77b8d9a20a3ece753d, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.799296 2016 raft_consensus.cc:729] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14817ea8f78646cf99b2c6bfe0da5a06, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.799418 2016 consensus_queue.cc:260] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799427 2014 consensus_queue.cc:260] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799453 2015 raft_consensus.cc:348] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799598 2015 raft_consensus.cc:374] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:57.799634 2015 raft_consensus.cc:729] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ad4ae809f046e8bb319c2935caafb6, State: Initialized, Role: FOLLOWER
I20260620 08:59:57.799693 2016 ts_tablet_manager.cc:1428] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:59:57.799700 2014 ts_tablet_manager.cc:1428] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:59:57.799773 2015 consensus_queue.cc:260] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:57.799897 2021 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:57.799923 2025 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:57.800040 2021 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:57.800050 2025 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:57.799996 2020 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:57.800105 2020 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:57.800308 2015 ts_tablet_manager.cc:1428] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:59:58.156258 2025 consensus_queue.cc:1035] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:58.159473 2022 consensus_queue.cc:1035] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:58.181313 2021 raft_consensus.cc:482] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:59:58.181473 2021 raft_consensus.cc:504] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:58.182723 2021 leader_election.cc:288] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2:45559), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:58.186640 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.183095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34303 (local address 127.0.98.2:45559)
0620 08:59:58.183349 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.183363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.183400 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:59:58.183560 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.183566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.183622 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.183693 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:58.184014 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.184571 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.185702 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.186097 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.186208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.186275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.186427 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.186504 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260620 08:59:58.186638 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.182985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45559 (local address 127.0.98.1:34303)
0620 08:59:58.183298 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:58.183314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:58.183447 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:58.183729 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:58.183734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:58.183760 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:58.183882 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.183891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.184671 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.184675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.185576 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.185584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.186214 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.186218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.186298 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.186321 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 08:59:58.186387 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:59:58.186471 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":42,"thread_start_us":72,"threads_started":1}
I20260620 08:59:58.186661 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.183013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36223 (local address 127.0.98.1:55551)
0620 08:59:58.183365 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:58.183396 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:58.183486 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:58.183820 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:58.183823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:58.183839 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:59:58.183934 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.183966 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.184791 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.184794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.185582 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.185590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.186247 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.186251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.186339 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.186359 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:59:58.186455 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 08:59:58.186524 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260620 08:59:58.186673 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.183163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55551 (local address 127.0.98.3:36223)
0620 08:59:58.183428 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.183432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.183449 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:59:58.183633 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.183635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.183699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.183786 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:59:58.184084 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.184677 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.185728 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.186108 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.186224 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.186284 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.186484 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.186555 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260620 08:59:58.187227 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08deb28cb0dd4eabaff00c04104eae3c" candidate_uuid: "33870f19f96e4a77b8d9a20a3ece753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" is_pre_election: true
I20260620 08:59:58.187433 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08deb28cb0dd4eabaff00c04104eae3c" candidate_uuid: "33870f19f96e4a77b8d9a20a3ece753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6" is_pre_election: true
I20260620 08:59:58.187466 1805 raft_consensus.cc:2454] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33870f19f96e4a77b8d9a20a3ece753d in term 0.
I20260620 08:59:58.187563 1945 raft_consensus.cc:2454] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33870f19f96e4a77b8d9a20a3ece753d in term 0.
I20260620 08:59:58.187852 1599 leader_election.cc:302] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14817ea8f78646cf99b2c6bfe0da5a06, 33870f19f96e4a77b8d9a20a3ece753d; no voters:
I20260620 08:59:58.188117 2021 raft_consensus.cc:2781] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:59:58.188181 2021 raft_consensus.cc:482] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:59:58.188230 2021 raft_consensus.cc:3037] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:58.189237 2021 raft_consensus.cc:504] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:58.189546 2021 leader_election.cc:288] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [CANDIDATE]: Term 1 election: Requested vote from peers 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2:45559), c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223)
I20260620 08:59:58.189941 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08deb28cb0dd4eabaff00c04104eae3c" candidate_uuid: "33870f19f96e4a77b8d9a20a3ece753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ad4ae809f046e8bb319c2935caafb6"
I20260620 08:59:58.189940 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08deb28cb0dd4eabaff00c04104eae3c" candidate_uuid: "33870f19f96e4a77b8d9a20a3ece753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14817ea8f78646cf99b2c6bfe0da5a06"
I20260620 08:59:58.190081 1945 raft_consensus.cc:3037] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:58.190088 1805 raft_consensus.cc:3037] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:58.190919 1945 raft_consensus.cc:2454] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33870f19f96e4a77b8d9a20a3ece753d in term 1.
I20260620 08:59:58.191079 1805 raft_consensus.cc:2454] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33870f19f96e4a77b8d9a20a3ece753d in term 1.
I20260620 08:59:58.191236 1596 leader_election.cc:302] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33870f19f96e4a77b8d9a20a3ece753d, c7ad4ae809f046e8bb319c2935caafb6; no voters:
I20260620 08:59:58.191390 2021 raft_consensus.cc:2781] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:58.191594 2021 raft_consensus.cc:686] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [term 1 LEADER]: Becoming Leader. State: Replica: 33870f19f96e4a77b8d9a20a3ece753d, State: Running, Role: LEADER
I20260620 08:59:58.191865 2021 consensus_queue.cc:237] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } }
I20260620 08:59:58.193483 2063 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: TxnStatusTablet state changed. Reason: New leader 33870f19f96e4a77b8d9a20a3ece753d. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.193594 2063 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:59:58.193838 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:59:58.193987 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:59:58.194653 1507 catalog_manager.cc:5495] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d reported cstate change: term changed from 0 to 1, leader changed from <none> to 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260620 08:59:58.251348 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.247168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.98.1:34411)
0620 08:59:58.247432 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.247440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.247588 (+ 148us) server_negotiation.cc:407] Connection header received
0620 08:59:58.247785 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.247787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.247841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.247937 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:58.248208 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.248759 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.249933 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.250153 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.250225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.250559 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.250977 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.251073 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.251174 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.251219 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
I20260620 08:59:58.251392 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.247032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34411 (local address 127.0.0.1:37838)
0620 08:59:58.247498 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:58.247515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:58.247639 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:58.247977 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:58.247980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:58.247995 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:59:58.248103 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.248109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.248864 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.248868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.249818 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.249828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.250256 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.250260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.250370 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.250424 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.251073 (+ 649us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.251086 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:59:58.251163 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:59:58.251250 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260620 08:59:58.254419 1945 raft_consensus.cc:1261] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Refusing update from remote peer 33870f19f96e4a77b8d9a20a3ece753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:58.254583 1805 raft_consensus.cc:1261] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Refusing update from remote peer 33870f19f96e4a77b8d9a20a3ece753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:59:58.255069 2063 consensus_queue.cc:1035] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:58.255259 2021 consensus_queue.cc:1035] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:59:58.256937 2020 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: TxnStatusTablet state changed. Reason: New leader 33870f19f96e4a77b8d9a20a3ece753d. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.256965 2022 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: TxnStatusTablet state changed. Reason: New leader 33870f19f96e4a77b8d9a20a3ece753d. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.257072 2020 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:58.257082 2022 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:58.258879 2021 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.259004 2021 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:59:58.259142 2021 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.259234 2021 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 33870f19f96e4a77b8d9a20a3ece753d: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:59:58.259418 2020 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.259512 2020 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P c7ad4ae809f046e8bb319c2935caafb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:58.259598 2022 tablet_replica.cc:439] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33870f19f96e4a77b8d9a20a3ece753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33870f19f96e4a77b8d9a20a3ece753d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34411 } } peers { permanent_uuid: "14817ea8f78646cf99b2c6bfe0da5a06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45559 } } peers { permanent_uuid: "c7ad4ae809f046e8bb319c2935caafb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36223 } } }
I20260620 08:59:58.259693 2022 tablet_replica.cc:442] T 08deb28cb0dd4eabaff00c04104eae3c P 14817ea8f78646cf99b2c6bfe0da5a06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:59:58.260805 2055 mvcc.cc:205] Tried to move back new op lower bound from 7298850808845012992 to 7298850808596111360. Current Snapshot: MvccSnapshot[applied={T|T < 7298850808845012992 or (T in {7298850808845012992})}]
I20260620 08:59:58.275419 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.271999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53967 (local address 127.0.98.1:34411)
0620 08:59:58.272136 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.272139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.272250 (+ 111us) server_negotiation.cc:407] Connection header received
0620 08:59:58.272403 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.272405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.272484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.272559 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:58.272797 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.273291 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.274473 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.274683 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.274750 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.275055 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.275116 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.275163 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.275276 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.275319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:59:58.275492 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.271874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34411 (local address 127.0.98.2:53967)
0620 08:59:58.272166 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:58.272181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:58.272282 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:58.272582 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:58.272585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:58.272594 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:59:58.272697 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.272703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.273389 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.273393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.274357 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.274368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.274786 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.274789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.274877 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.274928 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.275189 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.275195 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:59:58.275263 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:59:58.275345 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260620 08:59:58.301312 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.297198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52417 (local address 127.0.98.2:45559)
0620 08:59:58.297312 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.297316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.297358 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:59:58.297404 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.297407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.297460 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.297542 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:59:58.297735 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.298565 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.300102 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.300332 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.300412 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.300912 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.300993 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.301052 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.301167 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.301211 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:59:58.301494 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.296910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45559 (local address 127.0.98.1:52417)
0620 08:59:58.297212 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:58.297227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:58.297306 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:58.297546 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:58.297548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:58.297553 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:59:58.297639 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.297642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.298731 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.298735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.299743 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:58.299791 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.300439 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.300556 (+ 117us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:58.300726 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.300781 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:59:58.301078 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:59:58.301087 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 08:59:58.301162 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:59:58.301244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
I20260620 08:59:58.328899 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:59:58.329058 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:59:58.331329 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90c8fbf8b9194f2994706343d6177258: Sending DeleteTablet for 3 replicas of tablet 98c364ff8111471192ab641fa10fad07
I20260620 08:59:58.331946 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c364ff8111471192ab641fa10fad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38116
I20260620 08:59:58.332238 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c364ff8111471192ab641fa10fad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38312
I20260620 08:59:58.332316 2082 tablet_replica.cc:330] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: stopping tablet replica
I20260620 08:59:58.332470 2082 raft_consensus.cc:2229] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:59:58.332530 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c364ff8111471192ab641fa10fad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:59:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 08:59:58.332598 2082 raft_consensus.cc:2258] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:59:58.332808 2084 tablet_replica.cc:330] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: stopping tablet replica
I20260620 08:59:58.332926 2084 raft_consensus.cc:2229] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:59:58.333123 2084 raft_consensus.cc:2258] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:59:58.333272 2082 ts_tablet_manager.cc:1905] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:58.333714 2084 ts_tablet_manager.cc:1905] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:58.334575 2083 tablet_replica.cc:330] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: stopping tablet replica
I20260620 08:59:58.334775 2083 raft_consensus.cc:2229] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:59:58.335253 2083 raft_consensus.cc:2258] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:59:58.335942 2083 ts_tablet_manager.cc:1905] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:59:58.336302 2082 ts_tablet_manager.cc:1918] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:59:58.336396 2082 log.cc:1196] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/98c364ff8111471192ab641fa10fad07
I20260620 08:59:58.336516 2084 ts_tablet_manager.cc:1918] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:59:58.336602 2084 log.cc:1196] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/98c364ff8111471192ab641fa10fad07
I20260620 08:59:58.336710 2082 ts_tablet_manager.cc:1939] T 98c364ff8111471192ab641fa10fad07 P 33870f19f96e4a77b8d9a20a3ece753d: Deleting consensus metadata
I20260620 08:59:58.336874 2084 ts_tablet_manager.cc:1939] T 98c364ff8111471192ab641fa10fad07 P 14817ea8f78646cf99b2c6bfe0da5a06: Deleting consensus metadata
I20260620 08:59:58.337447 1493 catalog_manager.cc:4841] TS 33870f19f96e4a77b8d9a20a3ece753d (127.0.98.1:34411): tablet 98c364ff8111471192ab641fa10fad07 (table test-table [id=b45200273118468e8af061cc6c683975]) successfully deleted
I20260620 08:59:58.337447 1495 catalog_manager.cc:4841] TS 14817ea8f78646cf99b2c6bfe0da5a06 (127.0.98.2:45559): tablet 98c364ff8111471192ab641fa10fad07 (table test-table [id=b45200273118468e8af061cc6c683975]) successfully deleted
I20260620 08:59:58.339179 2083 ts_tablet_manager.cc:1918] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:59:58.339259 2083 log.cc:1196] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/98c364ff8111471192ab641fa10fad07
I20260620 08:59:58.339514 2083 ts_tablet_manager.cc:1939] T 98c364ff8111471192ab641fa10fad07 P c7ad4ae809f046e8bb319c2935caafb6: Deleting consensus metadata
I20260620 08:59:58.340229 1492 catalog_manager.cc:4841] TS c7ad4ae809f046e8bb319c2935caafb6 (127.0.98.3:36223): tablet 98c364ff8111471192ab641fa10fad07 (table test-table [id=b45200273118468e8af061cc6c683975]) successfully deleted
Jun 20 08:59:58 dist-test-slave-scsf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 08:59:58.350570 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.334322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.98.1:34411)
0620 08:59:58.334428 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.334433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.340342 (+ 5909us) server_negotiation.cc:407] Connection header received
0620 08:59:58.340520 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.340525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.340585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.340654 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:58.341515 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.342009 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.343075 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.343272 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.343343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.346704 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:58.346713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:58.346726 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:58.346749 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:58.348741 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:58.349196 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:58.349202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:58.349203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:58.349292 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:58.349576 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:58.349580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:58.349581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:58.349857 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:58.350078 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.350241 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.350383 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:59:58.351025 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37852
I20260620 08:59:58.361399 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:58.352786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.0.98.62:44311)
0620 08:59:58.352995 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:58.352999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:58.353123 (+ 124us) server_negotiation.cc:407] Connection header received
0620 08:59:58.353243 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:58.353245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:58.353284 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:58.353355 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:58.354129 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.354757 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.355824 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:58.356031 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:58.356152 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:58.357622 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:58.357625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:58.357627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:58.357647 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:58.359163 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:58.360001 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:58.360005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:58.360007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:58.360110 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:58.360405 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:58.360408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:58.360409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:58.360709 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:58.360864 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:58.361118 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 08:59:58.361243 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":54,"threads_started":1}
W20260620 08:59:58.361802 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55652
I20260620 08:59:58.362919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1580
I20260620 08:59:58.369849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1720
I20260620 08:59:58.376713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1860
I20260620 08:59:58.382190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 1476
2026-06-20T08:59:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3382 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:59:58 dist-test-slave-scsf krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:59:58 dist-test-slave-scsf krb5kdc[2092](info): set up 2 sockets
Jun 20 08:59:58 dist-test-slave-scsf krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260620 08:59:59.420284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 08:59:59 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 08:59:59.537526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:59.644855 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:59.645159 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:59.645232 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:59.649158 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:59:59.649240 2105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:59:59.649271 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:59.649312 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:59.649353 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:59:59.649385 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:59:59.654569 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:59.655661 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:59.656766 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:59.665067 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:59.665077 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:59.665067 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:59.665365 2105 server_base.cc:1034] running on GCE node
I20260620 08:59:59.666126 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:59.667167 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:59.668352 2105 hybrid_clock.cc:648] HybridClock initialized: now 1781945999668340 us; error 24 us; skew 500 ppm
Jun 20 08:59:59 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:59.675665 2105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 08:59:59.676633 2105 webserver.cc:460] Webserver started at http://127.0.98.62:34617/ using document root <none> and password file <none>
I20260620 08:59:59.677192 2105 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:59.677258 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:59.677517 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:59.679561 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "df224e0603634c8db9011e14b5a9976e"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "6eb263019672f71f4bf41e9d88d01fa3"
server_key_iv: "d9d9f2ca635acb97f9b44f71b2367a13"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.680111 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "df224e0603634c8db9011e14b5a9976e"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "6eb263019672f71f4bf41e9d88d01fa3"
server_key_iv: "d9d9f2ca635acb97f9b44f71b2367a13"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.683825 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 08:59:59.686224 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:59.687412 2105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:59.687561 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "df224e0603634c8db9011e14b5a9976e"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "6eb263019672f71f4bf41e9d88d01fa3"
server_key_iv: "d9d9f2ca635acb97f9b44f71b2367a13"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.687667 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:59.705588 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:59.708438 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:59.708674 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:59.716526 2182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33717 every 8 connection(s)
I20260620 08:59:59.716534 2105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33717
I20260620 08:59:59.717551 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 08:59:59.720232 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:59:59.723358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2105
I20260620 08:59:59.723461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 08:59:59.723726 392 external_mini_cluster.cc:1428] Setting key 4498492bbc58dd3561de34b7a2fa3589
I20260620 08:59:59.726114 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Bootstrap starting.
I20260620 08:59:59.728688 2183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Neither blocks nor log segments found. Creating new log.
I20260620 08:59:59.729565 2183 log.cc:824] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:59:59.731781 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: No bootstrap required, opened a new log
Jun 20 08:59:59 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:59.735153 2183 raft_consensus.cc:348] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } }
I20260620 08:59:59.735397 2183 raft_consensus.cc:374] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:59:59.735486 2183 raft_consensus.cc:729] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df224e0603634c8db9011e14b5a9976e, State: Initialized, Role: FOLLOWER
I20260620 08:59:59.735913 2183 consensus_queue.cc:260] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } }
I20260620 08:59:59.736066 2183 raft_consensus.cc:388] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:59:59.736147 2183 raft_consensus.cc:482] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:59:59.736263 2183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:59:59.737192 2183 raft_consensus.cc:504] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } }
I20260620 08:59:59.737525 2183 leader_election.cc:302] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df224e0603634c8db9011e14b5a9976e; no voters:
I20260620 08:59:59.737874 2183 leader_election.cc:288] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:59:59.737912 2188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:59:59.738121 2188 raft_consensus.cc:686] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [term 1 LEADER]: Becoming Leader. State: Replica: df224e0603634c8db9011e14b5a9976e, State: Running, Role: LEADER
I20260620 08:59:59.738408 2188 consensus_queue.cc:237] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } }
I20260620 08:59:59.738618 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:59.725064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.0.98.62:33717)
0620 08:59:59.725458 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:59.725483 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:59.725518 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:59:59.726158 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:59.726221 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:59.726564 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:59.726831 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:59.727762 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.728870 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.730011 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.730623 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.730753 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:59.733447 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:59.733452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:59.733464 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:59.733490 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:59.735189 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.735688 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.735699 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.735703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.735857 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.736178 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.736181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.736183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.736698 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:59.736904 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:59.737181 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 08:59:59.737578 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260620 08:59:59.739137 2183 sys_catalog.cc:564] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:59:59.739600 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df224e0603634c8db9011e14b5a9976e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } } }
I20260620 08:59:59.739706 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [sys.catalog]: This master's current role is: LEADER
I20260620 08:59:59.739989 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df224e0603634c8db9011e14b5a9976e. Latest consensus state: current_term: 1 leader_uuid: "df224e0603634c8db9011e14b5a9976e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df224e0603634c8db9011e14b5a9976e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33717 } } }
I20260620 08:59:59.740079 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e [sys.catalog]: This master's current role is: LEADER
W20260620 08:59:59.743636 2203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:59:59.743722 2203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:59:59.743795 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:59:59.744689 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:59:59.749727 2192 catalog_manager.cc:1347] Generated new cluster ID: b6ae4fa331dc4e518c1a4ad1d2400438
I20260620 08:59:59.749791 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:59:59.770093 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:59:59.771397 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:59:59.777895 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Generated new TSK 0
I20260620 08:59:59.778581 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 08:59:59.806921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33717
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:59:59.917968 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:59:59.918277 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:59:59.918356 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:59:59.922436 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:59:59.922516 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:59:59.922590 2209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:59:59.922634 2209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:59:59.922698 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 08:59:59.927814 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 08:59:59.928947 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:59:59.929967 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:59:59.939476 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:59.939597 2209 server_base.cc:1034] running on GCE node
W20260620 08:59:59.939561 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:59:59.939560 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:59:59.940160 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:59:59.940905 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:59:59.942093 2209 hybrid_clock.cc:648] HybridClock initialized: now 1781945999942082 us; error 31 us; skew 500 ppm
Jun 20 08:59:59 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:59:59.944744 2209 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 08:59:59.945569 2209 webserver.cc:460] Webserver started at http://127.0.98.1:41015/ using document root <none> and password file <none>
I20260620 08:59:59.946075 2209 fs_manager.cc:359] Metadata directory not provided
I20260620 08:59:59.946131 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:59:59.946286 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:59:59.948161 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd97d732584249de9181b33bba844cfd"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "b1c2deb520fa49c290f829a08ac9ca2d"
server_key_iv: "c64d6a55bc12bc1b04f748f0a2e5a84a"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.948715 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd97d732584249de9181b33bba844cfd"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "b1c2deb520fa49c290f829a08ac9ca2d"
server_key_iv: "c64d6a55bc12bc1b04f748f0a2e5a84a"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.952105 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 08:59:59.954519 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:59.955682 2209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:59:59.955811 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "bd97d732584249de9181b33bba844cfd"
format_stamp: "Formatted at 2026-06-20 08:59:59 on dist-test-slave-scsf"
server_key: "b1c2deb520fa49c290f829a08ac9ca2d"
server_key_iv: "c64d6a55bc12bc1b04f748f0a2e5a84a"
server_key_version: "encryptionkey@0"
I20260620 08:59:59.955919 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:59:59.966707 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:59:59.969760 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:59:59.969961 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:59:59.971395 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:59:59.971467 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:59:59.971534 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:59:59.971586 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:59:59 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 08:59:59.982657 2209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37195
I20260620 08:59:59.982741 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37195 every 8 connection(s)
I20260620 08:59:59.983687 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 08:59:59.985368 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:59.972191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57697 (local address 127.0.98.62:33717)
0620 08:59:59.972317 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:59.972321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:59.973420 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 08:59:59.974344 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:59.974347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:59.974404 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:59.974485 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:59.975841 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.976507 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.977908 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.978334 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.978466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:59.981470 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:59.981474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:59.981476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:59.981500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:59.983088 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.983683 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.983686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.983687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.983766 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.984219 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.984223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.984224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.984496 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:59.984603 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:59.985080 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 08:59:59.985201 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:59:59.986097 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:59.972766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.1:57697)
0620 08:59:59.973277 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:59.973318 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:59.974136 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:59.974666 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:59.974678 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:59.975168 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:59.975694 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:59.975708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.976661 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.976664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:59.977681 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:59.977690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.978512 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.978518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:59.978754 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:59.979376 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:59.979394 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:59.981308 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:59.983195 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:59.983200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:59.983214 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:59.983554 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:59.983907 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:59.983911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:59.983913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:59.984092 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:59.984605 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:59.984612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:59.984842 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 08:59:59.985025 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 08:59:59.985334 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260620 08:59:59.987285 2340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33717
I20260620 08:59:59.987421 2340 heartbeater.cc:464] Registering TS with master...
I20260620 08:59:59.987949 2340 heartbeater.cc:511] Master 127.0.98.62:33717 requested a full tablet report, sending...
I20260620 08:59:59.989595 2136 ts_manager.cc:194] Registered new tserver with Master: bd97d732584249de9181b33bba844cfd (127.0.98.1:37195)
I20260620 08:59:59.991070 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57697
I20260620 08:59:59.993605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2209
I20260620 08:59:59.993719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 08:59:59.994046 392 external_mini_cluster.cc:1428] Setting key 9be8f49f0ad063e8bad2038aa0e3e007
I20260620 08:59:59.997010 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:59.989619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37487 (local address 127.0.98.62:33717)
0620 08:59:59.989730 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 08:59:59.989735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:59:59.989901 (+ 166us) server_negotiation.cc:407] Connection header received
0620 08:59:59.990068 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:59:59.990071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:59:59.990118 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:59:59.990225 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:59:59.990692 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.991327 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.992391 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.992712 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.992814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:59.993997 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:59:59.993999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:59:59.994001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:59:59.994021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:59:59.995520 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.995948 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.995951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.995952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.996014 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:59:59.996300 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:59:59.996302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:59:59.996303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:59:59.996438 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:59:59.996579 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 08:59:59.996830 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 08:59:59.996909 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":40}
I20260620 08:59:59.997256 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:59:59.989547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.1:37487)
0620 08:59:59.989823 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 08:59:59.989834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:59:59.989940 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:59:59.990241 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:59:59.990246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:59:59.990459 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:59:59.990577 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:59.990584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.991461 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.991464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:59.992282 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:59:59.992291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:59:59.992815 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:59:59.992818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:59:59.992884 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:59:59.993423 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:59:59.993441 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:59:59.993873 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:59:59.995617 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:59.995620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:59.995622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:59.995853 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:59.996101 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:59:59.996106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:59:59.996111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:59:59.996212 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:59:59.996565 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:59:59.996571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:59:59.996677 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 08:59:59.996783 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:59:59.996940 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:00.021103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33717
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:00.134102 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:00.134397 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:00.134447 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:00.139459 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:00.139593 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:00.139640 2349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:00:00.139662 2349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:00:00.139696 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:00.145443 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:00.146559 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:00.147642 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:00.157786 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:00.157779 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:00.157805 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:00.158110 2349 server_base.cc:1034] running on GCE node
I20260620 09:00:00.158738 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:00.159472 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:00.160665 2349 hybrid_clock.cc:648] HybridClock initialized: now 1781946000160659 us; error 53 us; skew 500 ppm
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:00.163545 2349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:00.164506 2349 webserver.cc:460] Webserver started at http://127.0.98.2:41735/ using document root <none> and password file <none>
I20260620 09:00:00.165180 2349 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:00.165251 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:00.165424 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:00.167572 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb0508912a224fe48abe51b735e32d8a"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "d4f5dfa90706134f28492d09cc43272b"
server_key_iv: "87718702eefe1ec9fd6e3843c551850d"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.168150 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb0508912a224fe48abe51b735e32d8a"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "d4f5dfa90706134f28492d09cc43272b"
server_key_iv: "87718702eefe1ec9fd6e3843c551850d"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.172093 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 09:00:00.174757 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:00.176095 2349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:00.176220 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "cb0508912a224fe48abe51b735e32d8a"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "d4f5dfa90706134f28492d09cc43272b"
server_key_iv: "87718702eefe1ec9fd6e3843c551850d"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.176326 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:00.191061 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:00.194423 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:00.194607 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:00.196053 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:00.196108 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:00.196151 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:00.196218 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:00.207388 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32941
I20260620 09:00:00.207471 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32941 every 8 connection(s)
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:00.208630 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:00.212476 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.197078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47881 (local address 127.0.98.62:33717)
0620 09:00:00.197213 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.197217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.198390 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 09:00:00.199534 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.199538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.199607 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.199724 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.201649 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.202515 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.204559 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.204780 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.204870 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.208295 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.208299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.208301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.208330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.210246 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.210808 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.210812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.210814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.210908 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.211292 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.211297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.211299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.211496 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.211617 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.212141 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.212286 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:00:00.213348 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.197696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.2:47881)
0620 09:00:00.198217 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.198263 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.199296 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.199977 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.199991 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.200602 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:00.201441 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.201467 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.202689 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.202696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.204338 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.204350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.204961 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.204967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.205256 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.206089 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:00.206119 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:00.208070 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:00.210364 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.210369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.210381 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.210671 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.211013 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.211016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.211018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.211187 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.211615 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:00.211624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:00.211890 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.212103 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.212483 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260620 09:00:00.214633 2480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33717
I20260620 09:00:00.214839 2480 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:00.215366 2480 heartbeater.cc:511] Master 127.0.98.62:33717 requested a full tablet report, sending...
I20260620 09:00:00.216423 2136 ts_manager.cc:194] Registered new tserver with Master: cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941)
I20260620 09:00:00.217171 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47881
I20260620 09:00:00.217990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2349
I20260620 09:00:00.218078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:00.218384 392 external_mini_cluster.cc:1428] Setting key fedff5832d2c396502630723e6690d01
I20260620 09:00:00.227556 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.217089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46715 (local address 127.0.98.62:33717)
0620 09:00:00.217228 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.217233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.217473 (+ 240us) server_negotiation.cc:407] Connection header received
0620 09:00:00.217588 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.217591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.217640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.217703 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.220566 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.221431 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.222317 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.222549 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.222677 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.224197 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.224201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.224203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.224228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.225738 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.226302 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.226304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.226306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.226381 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.226694 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.226696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.226698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.226911 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.226986 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.227329 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.227427 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:00:00.227671 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.217012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.2:46715)
0620 09:00:00.217373 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.217391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.217502 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.217742 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.217745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.219494 (+ 1749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:00.219614 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.219622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.221584 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.221601 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.222174 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.222183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.222689 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.222693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.222800 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.223480 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:00.223502 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:00.224030 (+ 528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:00.225872 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.225877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.225880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.226197 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.226481 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.226484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.226486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.226591 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.227014 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:00.227017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:00.227141 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.227270 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.227460 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:00.245518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33717
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:00.361893 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:00.362186 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:00.362236 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:00.366608 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:00.366689 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:00.366732 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:00:00.366747 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:00:00.366781 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:00.372422 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:00.373545 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:00.374548 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:00.384603 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:00.384591 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:00.384591 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:00.384907 2489 server_base.cc:1034] running on GCE node
I20260620 09:00:00.385383 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:00.386046 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:00.387217 2489 hybrid_clock.cc:648] HybridClock initialized: now 1781946000387183 us; error 56 us; skew 500 ppm
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:00.390214 2489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:00.391304 2489 webserver.cc:460] Webserver started at http://127.0.98.3:34447/ using document root <none> and password file <none>
I20260620 09:00:00.391969 2489 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:00.392036 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:00.392318 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:00.394627 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2d3d58f0a0a435fb77814f962844c2e"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "5ee9e0105baa6cd066f6186e5995e1f0"
server_key_iv: "5dd338b85cd39142e35b4c24dcbae34f"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.395195 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d3d58f0a0a435fb77814f962844c2e"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "5ee9e0105baa6cd066f6186e5995e1f0"
server_key_iv: "5dd338b85cd39142e35b4c24dcbae34f"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.399220 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:00:00.402225 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:00.403671 2489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:00:00.403851 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "b2d3d58f0a0a435fb77814f962844c2e"
format_stamp: "Formatted at 2026-06-20 09:00:00 on dist-test-slave-scsf"
server_key: "5ee9e0105baa6cd066f6186e5995e1f0"
server_key_iv: "5dd338b85cd39142e35b4c24dcbae34f"
server_key_version: "encryptionkey@0"
I20260620 09:00:00.403976 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:00.422972 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:00.426630 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:00.426862 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:00.428361 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:00.428434 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:00.428583 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:00.428633 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:00.439711 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33157
I20260620 09:00:00.439734 2619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33157 every 8 connection(s)
I20260620 09:00:00.440874 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:00.442426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2489
I20260620 09:00:00.442512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:00.442802 392 external_mini_cluster.cc:1428] Setting key 74c3ca3a718046fa4cdc324473bfcbda
I20260620 09:00:00.443635 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.429356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47397 (local address 127.0.98.62:33717)
0620 09:00:00.429492 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.429496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.430625 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 09:00:00.431712 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.431716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.431785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.431891 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.433574 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.434524 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.435803 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.436207 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.436329 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.439379 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.439383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.439387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.439421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.441285 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.441895 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.441898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.441900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.442011 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.442365 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.442369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.442371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.442639 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.442747 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.443340 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.443476 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:00:00.444576 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.429931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.3:47397)
0620 09:00:00.430444 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.430489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.431472 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.432160 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.432171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.432692 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:00.433357 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.433378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.434676 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.434680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.435690 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.435700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.436393 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.436398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.436653 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.437315 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:00.437331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:00.439174 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:00.441439 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.441445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.441473 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.441777 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.442116 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.442118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.442119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.442257 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.442756 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:00.442765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:00.443083 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.443280 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.443635 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260620 09:00:00.445762 2620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33717
I20260620 09:00:00.445927 2620 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:00.446455 2620 heartbeater.cc:511] Master 127.0.98.62:33717 requested a full tablet report, sending...
I20260620 09:00:00.447533 2136 ts_manager.cc:194] Registered new tserver with Master: b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:00.448165 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47397
I20260620 09:00:00.456213 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.448134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59163 (local address 127.0.98.62:33717)
0620 09:00:00.448277 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.448284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.448508 (+ 224us) server_negotiation.cc:407] Connection header received
0620 09:00:00.448640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.448642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.448680 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.448742 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.450379 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.450872 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.451647 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.451849 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.451925 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.453140 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.453144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.453145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.453168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.454629 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.455020 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.455023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.455024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.455094 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.455317 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.455318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.455320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.455488 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.455566 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.455917 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.456055 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:00.456214 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.448082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.98.3:59163)
0620 09:00:00.448393 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.448410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.448551 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.449990 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.449992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.450176 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:00.450270 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.450276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.451008 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.451010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.451551 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.451558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.451946 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.451949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.452036 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.452619 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:00.452630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:00.453007 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:00.454731 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.454734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.454736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.454929 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.455169 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.455171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.455173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.455245 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.455592 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:00.455594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:00.455701 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.455832 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.455984 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":165,"threads_started":1}
I20260620 09:00:00.457242 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:00.493963 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.482043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.0.98.62:33717)
0620 09:00:00.482209 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.482215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.482234 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:00:00.482294 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.482300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.482363 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.482490 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.483192 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.484137 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.485191 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.485606 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.485725 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.489679 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.489684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.489687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.489726 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.491585 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.492192 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.492198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.492200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.492307 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.492706 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.492710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.492712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.493035 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.493216 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.493572 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.493723 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:00:00.498023 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:00.503643 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:00.520624 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.515451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.98.2:32941)
0620 09:00:00.516172 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.516187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.516208 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:00:00.516279 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.516284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.516522 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.516675 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.516863 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517634 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.518795 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.519582 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.519686 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.519851 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.520327 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.520435 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":115,"threads_started":1}
I20260620 09:00:00.520617 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.514320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.0.1:58488)
0620 09:00:00.515264 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.515278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.515372 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.515766 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.515769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.515789 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.515926 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.515933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517874 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.517878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.519054 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.519063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.520074 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.520077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.520191 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.520242 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.520314 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.520468 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":46,"thread_start_us":103,"threads_started":1}
I20260620 09:00:00.520670 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.514761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.0.98.1:37195)
0620 09:00:00.515119 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.515126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.515347 (+ 221us) server_negotiation.cc:407] Connection header received
0620 09:00:00.515474 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.515479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.515626 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.516085 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.516226 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517450 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.519154 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.519967 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.520065 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.520211 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.520420 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.520542 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":88,"threads_started":1}
I20260620 09:00:00.521060 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.514285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.0.98.3:33157)
0620 09:00:00.515219 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.515227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.515247 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:00:00.515343 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.515348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.515561 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.515719 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.516211 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517450 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.519105 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.519892 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.520447 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.520651 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.520706 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.520905 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":96,"threads_started":1}
I20260620 09:00:00.521291 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.514225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33157 (local address 127.0.0.1:42108)
0620 09:00:00.514716 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.514770 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.514902 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.515726 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.515733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.515758 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.515925 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.515931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517576 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.517579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.518976 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.518987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.520017 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.520020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.520183 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.520225 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.520282 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.521149 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":831,"thread_start_us":82,"threads_started":1}
I20260620 09:00:00.521625 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.515006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.0.1:47248)
0620 09:00:00.516017 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.516032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.516104 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.516662 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.516665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.516682 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.516767 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.516776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.517732 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.517734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.518673 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.518683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.519914 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.519921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.520111 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.520164 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.520280 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.521495 (+ 1215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":55,"thread_start_us":118,"threads_started":1}
I20260620 09:00:00.524673 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 (DEFAULT_TABLE table=test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.524549 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 (DEFAULT_TABLE table=test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.525636 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d08d8ac60754b4589dc58990c7d5b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.525656 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d08d8ac60754b4589dc58990c7d5b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.526482 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 (DEFAULT_TABLE table=test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.527540 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d08d8ac60754b4589dc58990c7d5b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.533003 2643 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Bootstrap starting.
I20260620 09:00:00.534166 2644 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Bootstrap starting.
I20260620 09:00:00.534432 2645 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Bootstrap starting.
I20260620 09:00:00.535511 2643 tablet_bootstrap.cc:654] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.536033 2644 tablet_bootstrap.cc:654] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.536381 2643 log.cc:824] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:00.536765 2645 tablet_bootstrap.cc:654] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.536883 2644 log.cc:824] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:00.537563 2645 log.cc:824] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:00.538754 2643 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: No bootstrap required, opened a new log
I20260620 09:00:00.538928 2644 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: No bootstrap required, opened a new log
I20260620 09:00:00.538954 2643 ts_tablet_manager.cc:1397] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:00:00.539116 2644 ts_tablet_manager.cc:1397] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:00:00.539603 2645 tablet_bootstrap.cc:492] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: No bootstrap required, opened a new log
I20260620 09:00:00.539813 2645 ts_tablet_manager.cc:1397] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:00:00.542402 2643 raft_consensus.cc:348] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.542683 2643 raft_consensus.cc:374] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.542737 2643 raft_consensus.cc:729] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.543184 2643 consensus_queue.cc:260] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.543171 2644 raft_consensus.cc:348] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.543422 2644 raft_consensus.cc:374] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.543504 2644 raft_consensus.cc:729] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.543921 2644 consensus_queue.cc:260] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.544281 2643 ts_tablet_manager.cc:1428] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 09:00:00.544696 2620 heartbeater.cc:503] Master 127.0.98.62:33717 was elected leader, sending a full tablet report...
I20260620 09:00:00.544703 2480 heartbeater.cc:503] Master 127.0.98.62:33717 was elected leader, sending a full tablet report...
I20260620 09:00:00.544909 2644 ts_tablet_manager.cc:1428] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:00:00.544363 2645 raft_consensus.cc:348] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.545414 2645 raft_consensus.cc:374] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.545491 2645 raft_consensus.cc:729] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97d732584249de9181b33bba844cfd, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.545938 2645 consensus_queue.cc:260] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.546628 2340 heartbeater.cc:503] Master 127.0.98.62:33717 was elected leader, sending a full tablet report...
I20260620 09:00:00.546940 2645 ts_tablet_manager.cc:1428] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:00:00.593348 2651 raft_consensus.cc:482] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:00.593618 2651 raft_consensus.cc:504] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.594944 2651 leader_election.cc:288] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157), cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941)
I20260620 09:00:00.600312 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.595293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56497 (local address 127.0.98.3:33157)
0620 09:00:00.595543 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.595548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.595565 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:00:00.595735 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.595738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.595789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.595873 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.596224 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.597054 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.598990 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.599637 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.599754 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.599822 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.600099 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.600173 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 09:00:00.600312 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.595382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56657 (local address 127.0.98.2:32941)
0620 09:00:00.595579 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.595586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.595601 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:00.595734 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.595737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.595789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.595873 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.596224 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.597079 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.598990 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.599632 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.599754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.599822 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.600110 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.600176 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 09:00:00.600378 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.595247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.98.1:56657)
0620 09:00:00.595491 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.595508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.595623 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.595900 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.595902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.595938 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.596100 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.596107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.597181 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.597184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.598817 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.598830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.599770 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.599774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.599924 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.599978 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.600058 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.600165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":180,"spinlock_wait_cycles":60160,"thread_start_us":110,"threads_started":1}
I20260620 09:00:00.600606 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.595137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33157 (local address 127.0.98.1:56497)
0620 09:00:00.595491 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.595508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.595615 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.595900 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.595902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.595939 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.596102 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.596107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.597181 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.597184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.598817 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.598830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.599770 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.599774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.599919 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.599989 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.600058 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.600165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":251,"spinlock_wait_cycles":40704,"thread_start_us":137,"threads_started":1}
I20260620 09:00:00.601243 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d08d8ac60754b4589dc58990c7d5b77" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e" is_pre_election: true
I20260620 09:00:00.601249 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d08d8ac60754b4589dc58990c7d5b77" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a" is_pre_election: true
I20260620 09:00:00.601567 2574 raft_consensus.cc:2454] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:00.601570 2434 raft_consensus.cc:2454] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:00.602097 2225 leader_election.cc:302] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, bd97d732584249de9181b33bba844cfd, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:00.602389 2651 raft_consensus.cc:2781] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:00.602463 2651 raft_consensus.cc:482] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:00.602496 2651 raft_consensus.cc:3037] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.603577 2651 raft_consensus.cc:504] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.603937 2651 leader_election.cc:288] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Requested vote from peers b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157), cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941)
I20260620 09:00:00.604338 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d08d8ac60754b4589dc58990c7d5b77" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a"
I20260620 09:00:00.604338 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d08d8ac60754b4589dc58990c7d5b77" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e"
I20260620 09:00:00.604503 2574 raft_consensus.cc:3037] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.604490 2434 raft_consensus.cc:3037] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.605813 2574 raft_consensus.cc:2454] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:00.605813 2434 raft_consensus.cc:2454] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:00.606163 2228 leader_election.cc:302] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, bd97d732584249de9181b33bba844cfd; no voters:
I20260620 09:00:00.606362 2651 raft_consensus.cc:2781] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:00.606619 2651 raft_consensus.cc:686] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 1 LEADER]: Becoming Leader. State: Replica: bd97d732584249de9181b33bba844cfd, State: Running, Role: LEADER
I20260620 09:00:00.606961 2651 consensus_queue.cc:237] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } }
I20260620 09:00:00.610483 2137 catalog_manager.cc:5495] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to bd97d732584249de9181b33bba844cfd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd97d732584249de9181b33bba844cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:00:00 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781945999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:00.643260 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.634289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33717 (local address 127.0.0.1:51540)
0620 09:00:00.634618 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.634635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.634752 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.634988 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.634990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.635416 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:00.635508 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.635513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.636217 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.636220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.636875 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.636883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.637270 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.637273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.637347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.637882 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:00.637894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:00.639529 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:00.641405 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.641413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.641418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.641671 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.641919 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:00.641921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:00.641922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:00.642006 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:00.642458 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:00.642467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:00.642628 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.642785 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.642993 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":152,"threads_started":1}
I20260620 09:00:00.643307 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.634326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.0.98.62:33717)
0620 09:00:00.634440 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.634444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.634717 (+ 273us) server_negotiation.cc:407] Connection header received
0620 09:00:00.634828 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.634830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.634868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.634950 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:00.635620 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.636124 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.636970 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.637173 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.637250 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.639704 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:00.639707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:00.639709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:00.639730 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:00.641286 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.641763 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.641765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.641766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.641837 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:00.642083 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:00.642085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:00.642086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:00.642331 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:00.642450 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.642841 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.643100 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":44}
I20260620 09:00:00.647480 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:00:00.648252 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:00.653132 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b600d489c8a4c0c95dd4297d6256899]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:00.653402 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b600d489c8a4c0c95dd4297d6256899]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:00.653384 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b600d489c8a4c0c95dd4297d6256899]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:00.653981 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.653982 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.654078 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e2cbe8d4434dcd9dc8e8784ac9611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.656810 2644 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: Bootstrap starting.
I20260620 09:00:00.656915 2645 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: Bootstrap starting.
I20260620 09:00:00.657259 2643 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: Bootstrap starting.
I20260620 09:00:00.657881 2644 tablet_bootstrap.cc:654] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.658068 2643 tablet_bootstrap.cc:654] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.658207 2645 tablet_bootstrap.cc:654] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.659535 2644 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: No bootstrap required, opened a new log
I20260620 09:00:00.659528 2645 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: No bootstrap required, opened a new log
I20260620 09:00:00.659622 2645 ts_tablet_manager.cc:1397] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:00.659622 2644 ts_tablet_manager.cc:1397] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:00.659745 2643 tablet_bootstrap.cc:492] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: No bootstrap required, opened a new log
I20260620 09:00:00.659830 2643 ts_tablet_manager.cc:1397] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:00:00.660069 2645 raft_consensus.cc:348] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660080 2644 raft_consensus.cc:348] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660187 2645 raft_consensus.cc:374] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.660203 2644 raft_consensus.cc:374] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.660229 2645 raft_consensus.cc:729] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97d732584249de9181b33bba844cfd, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.660241 2644 raft_consensus.cc:729] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.660229 2643 raft_consensus.cc:348] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660342 2643 raft_consensus.cc:374] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.660353 2645 consensus_queue.cc:260] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660353 2644 consensus_queue.cc:260] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660401 2643 raft_consensus.cc:729] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.660609 2643 consensus_queue.cc:260] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.660655 2645 ts_tablet_manager.cc:1428] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:00.660677 2651 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.660799 2651 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.660867 2643 ts_tablet_manager.cc:1428] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:00.661111 2644 ts_tablet_manager.cc:1428] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:00.661054 2650 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.661188 2650 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.661412 2649 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.661558 2649 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 09:00:00.692299 2621 tablet.cc:2369] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:00.710186 2481 tablet.cc:2369] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:00.735158 2341 tablet.cc:2369] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:00.756285 2650 raft_consensus.cc:482] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:00.756422 2650 raft_consensus.cc:504] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.757730 2650 leader_election.cc:288] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd97d732584249de9181b33bba844cfd (127.0.98.1:37195), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:00.761929 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.758061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33157 (local address 127.0.98.2:33773)
0620 09:00:00.758402 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.758423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.758550 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.758832 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.758836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.758858 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.758990 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.758997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.759691 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.759695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.760767 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.760777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.761469 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.761474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.761580 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.761603 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.761683 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.761745 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":18,"thread_start_us":88,"threads_started":1}
I20260620 09:00:00.761930 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.758176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33773 (local address 127.0.98.3:33157)
0620 09:00:00.758312 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.758319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.758535 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:00:00.758682 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.758685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.758729 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.758805 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.759103 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.759593 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.760891 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.761340 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.761423 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.761477 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.761714 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.761803 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:00:00.762008 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.758043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49429 (local address 127.0.98.1:37195)
0620 09:00:00.758175 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.758180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.758355 (+ 175us) server_negotiation.cc:407] Connection header received
0620 09:00:00.758506 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.758511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.758570 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.758649 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:00.758936 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.759704 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.761013 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.761453 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.761559 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.761615 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.761809 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.761877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:00:00.761974 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.757934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.98.2:49429)
0620 09:00:00.758248 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.758262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.758389 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.758676 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.758679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.758698 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:00.758830 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.758837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.759822 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.759824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.760875 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.760886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.761576 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.761579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.761675 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.761703 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.761778 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.761843 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":36,"thread_start_us":74,"threads_started":1}
I20260620 09:00:00.762631 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e2cbe8d4434dcd9dc8e8784ac9611c" candidate_uuid: "cb0508912a224fe48abe51b735e32d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e" is_pre_election: true
I20260620 09:00:00.762790 2574 raft_consensus.cc:2454] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0508912a224fe48abe51b735e32d8a in term 0.
I20260620 09:00:00.762758 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e2cbe8d4434dcd9dc8e8784ac9611c" candidate_uuid: "cb0508912a224fe48abe51b735e32d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd97d732584249de9181b33bba844cfd" is_pre_election: true
I20260620 09:00:00.763026 2294 raft_consensus.cc:2454] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0508912a224fe48abe51b735e32d8a in term 0.
I20260620 09:00:00.763150 2368 leader_election.cc:302] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:00.763459 2650 raft_consensus.cc:2781] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:00.763530 2650 raft_consensus.cc:482] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:00.763581 2650 raft_consensus.cc:3037] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.764737 2650 raft_consensus.cc:504] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.765134 2650 leader_election.cc:288] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [CANDIDATE]: Term 1 election: Requested vote from peers bd97d732584249de9181b33bba844cfd (127.0.98.1:37195), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:00.765443 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e2cbe8d4434dcd9dc8e8784ac9611c" candidate_uuid: "cb0508912a224fe48abe51b735e32d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd97d732584249de9181b33bba844cfd"
I20260620 09:00:00.765589 2294 raft_consensus.cc:3037] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.765560 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e2cbe8d4434dcd9dc8e8784ac9611c" candidate_uuid: "cb0508912a224fe48abe51b735e32d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e"
I20260620 09:00:00.765690 2574 raft_consensus.cc:3037] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.766676 2294 raft_consensus.cc:2454] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0508912a224fe48abe51b735e32d8a in term 1.
I20260620 09:00:00.766736 2574 raft_consensus.cc:2454] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0508912a224fe48abe51b735e32d8a in term 1.
I20260620 09:00:00.766971 2365 leader_election.cc:302] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd97d732584249de9181b33bba844cfd, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:00.767138 2650 raft_consensus.cc:2781] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:00.767359 2650 raft_consensus.cc:686] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [term 1 LEADER]: Becoming Leader. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Running, Role: LEADER
I20260620 09:00:00.767724 2650 consensus_queue.cc:237] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.769886 2667 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: TxnStatusTablet state changed. Reason: New leader cb0508912a224fe48abe51b735e32d8a. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.770015 2667 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:00.770288 2669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:00:00.770493 2669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:00:00.771308 2135 catalog_manager.cc:5495] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0508912a224fe48abe51b735e32d8a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:00.785079 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.781456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.0.98.2:32941)
0620 09:00:00.781594 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.781599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.781614 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:00.781800 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.781803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.781866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.781967 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:00.782211 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.782710 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.783554 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.783787 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.783903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.784202 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.784676 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.784795 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.784872 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.784924 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:00.785071 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.781366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.0.1:47258)
0620 09:00:00.781496 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.781512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.781640 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.781979 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.781982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.781989 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:00.782095 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.782101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.782818 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.782823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.783433 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.783441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.783902 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.783906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.783999 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.784051 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.784779 (+ 728us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.784795 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.784862 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.784943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:00:00.790246 2574 raft_consensus.cc:1261] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Refusing update from remote peer cb0508912a224fe48abe51b735e32d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:00.790212 2294 raft_consensus.cc:1261] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Refusing update from remote peer cb0508912a224fe48abe51b735e32d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:00.790855 2667 consensus_queue.cc:1035] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:00.791019 2650 consensus_queue.cc:1035] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:00.794389 2651 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: TxnStatusTablet state changed. Reason: New leader cb0508912a224fe48abe51b735e32d8a. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.794514 2651 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.798957 2649 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: TxnStatusTablet state changed. Reason: New leader cb0508912a224fe48abe51b735e32d8a. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.799088 2649 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.801000 2667 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.801133 2667 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:00.801721 2649 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.801831 2649 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P b2d3d58f0a0a435fb77814f962844c2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.803256 2651 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.803366 2651 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P bd97d732584249de9181b33bba844cfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:00.804776 2677 tablet_replica.cc:439] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb0508912a224fe48abe51b735e32d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } } }
I20260620 09:00:00.804960 2677 tablet_replica.cc:442] T 03e2cbe8d4434dcd9dc8e8784ac9611c P cb0508912a224fe48abe51b735e32d8a: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:00.811596 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.808014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.0.98.1:37195)
0620 09:00:00.808132 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.808136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.808149 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:00.808219 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.808222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.808292 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.808424 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:00.808707 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.809246 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.810124 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.810382 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.810490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.810783 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.811199 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.811329 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.811391 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.811445 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:00:00.823518 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.819904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45831 (local address 127.0.98.2:32941)
0620 09:00:00.820050 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.820055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.820211 (+ 156us) server_negotiation.cc:407] Connection header received
0620 09:00:00.820360 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.820362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.820404 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.820514 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:00.820757 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.821253 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.822506 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.822718 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.822793 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.823129 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.823196 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.823249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.823362 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.823408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:00:00.823597 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.819806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.98.1:45831)
0620 09:00:00.820140 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.820155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.820264 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.820521 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.820524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.820532 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:00.820672 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.820678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.821351 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.821355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.822388 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.822399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.822821 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.822825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.822918 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.822971 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.823275 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.823281 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.823347 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.823423 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":129,"threads_started":1}
I20260620 09:00:00.831099 2574 raft_consensus.cc:1261] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:00.831164 2434 raft_consensus.cc:1261] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:00.831604 2651 consensus_queue.cc:1035] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:00.831801 2654 consensus_queue.cc:1035] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:00.836549 2673 mvcc.cc:205] Tried to move back new op lower bound from 7298850819399061504 to 7298850818488913920. Current Snapshot: MvccSnapshot[applied={T|T < 7298850819399061504}]
I20260620 09:00:00.837373 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298850819399061504 to 7298850818488913920. Current Snapshot: MvccSnapshot[applied={T|T < 7298850819399061504}]
I20260620 09:00:00.839855 2672 mvcc.cc:205] Tried to move back new op lower bound from 7298850819399061504 to 7298850818488913920. Current Snapshot: MvccSnapshot[applied={T|T < 7298850819399061504}]
I20260620 09:00:00.857780 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.853637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50295 (local address 127.0.98.1:37195)
0620 09:00:00.853774 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:00.853778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:00.853951 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:00:00.854092 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:00.854098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:00.854160 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:00.854237 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:00.854517 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.855244 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.856524 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.856853 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.856953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.857290 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.857375 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.857419 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:00.857569 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 09:00:00.857624 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:00.858155 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:00.853297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.98.2:50295)
0620 09:00:00.853862 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:00.853877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:00.853988 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:00.854265 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:00.854268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:00.854289 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:00.854418 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.854424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.855349 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.855353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.856370 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:00.856380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:00.856981 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:00.856985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:00.857083 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:00.857137 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:00.857476 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:00.857481 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:00:00.857931 (+ 450us) client_negotiation.cc:240] Negotiation successful
0620 09:00:00.858011 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":115,"threads_started":1}
I20260620 09:00:00.888315 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:00.888597 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:00.891511 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Sending DeleteTablet for 3 replicas of tablet 4d08d8ac60754b4589dc58990c7d5b77
I20260620 09:00:00.892336 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42108
I20260620 09:00:00.892547 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58488
I20260620 09:00:00.893270 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d08d8ac60754b4589dc58990c7d5b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47248
I20260620 09:00:00.893335 2693 tablet_replica.cc:330] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: stopping tablet replica
I20260620 09:00:00.893738 2693 raft_consensus.cc:2229] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:00.894002 2693 raft_consensus.cc:2258] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:00.894553 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:00.895002 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:00.895907 2693 ts_tablet_manager.cc:1905] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:00.899230 2693 ts_tablet_manager.cc:1918] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:00.899425 2693 log.cc:1196] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/4d08d8ac60754b4589dc58990c7d5b77
I20260620 09:00:00.899820 2693 ts_tablet_manager.cc:1939] T 4d08d8ac60754b4589dc58990c7d5b77 P b2d3d58f0a0a435fb77814f962844c2e: Deleting consensus metadata
I20260620 09:00:00.899945 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 187449d10fd14507949363820ac7d67b (DEFAULT_TABLE table=test-table [id=4c2c51207e1f4a629b89fe574590c0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.900228 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187449d10fd14507949363820ac7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.901352 2124 catalog_manager.cc:4841] TS b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157): tablet 4d08d8ac60754b4589dc58990c7d5b77 (table test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]) successfully deleted
I20260620 09:00:00.902420 2643 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Bootstrap starting.
I20260620 09:00:00.903002 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 187449d10fd14507949363820ac7d67b (DEFAULT_TABLE table=test-table [id=4c2c51207e1f4a629b89fe574590c0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.903273 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187449d10fd14507949363820ac7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.903467 2643 tablet_bootstrap.cc:654] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.905040 2643 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: No bootstrap required, opened a new log
I20260620 09:00:00.905123 2643 ts_tablet_manager.cc:1397] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 09:00:00.905539 2643 raft_consensus.cc:348] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.905671 2643 raft_consensus.cc:374] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.905702 2643 raft_consensus.cc:729] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.905820 2643 consensus_queue.cc:260] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.905906 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 187449d10fd14507949363820ac7d67b (DEFAULT_TABLE table=test-table [id=4c2c51207e1f4a629b89fe574590c0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:00.906064 2643 ts_tablet_manager.cc:1428] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:00:00.906189 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187449d10fd14507949363820ac7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:00.908574 2644 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Bootstrap starting.
I20260620 09:00:00.908782 2645 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Bootstrap starting.
I20260620 09:00:00.909592 2644 tablet_bootstrap.cc:654] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.909705 2645 tablet_bootstrap.cc:654] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:00.910918 2644 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: No bootstrap required, opened a new log
I20260620 09:00:00.911002 2644 ts_tablet_manager.cc:1397] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:00.911425 2644 raft_consensus.cc:348] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.911531 2644 raft_consensus.cc:374] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.911561 2644 raft_consensus.cc:729] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.911691 2644 consensus_queue.cc:260] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.911948 2644 ts_tablet_manager.cc:1428] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:00.912147 2694 tablet_replica.cc:330] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: stopping tablet replica
I20260620 09:00:00.912505 2694 raft_consensus.cc:2229] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:00.912873 2694 raft_consensus.cc:2258] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:00.913226 2695 tablet_replica.cc:330] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: stopping tablet replica
I20260620 09:00:00.913905 2695 raft_consensus.cc:2229] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:00.914120 2695 raft_consensus.cc:2258] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:00.915561 2695 ts_tablet_manager.cc:1905] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:00.915669 2645 tablet_bootstrap.cc:492] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: No bootstrap required, opened a new log
I20260620 09:00:00.915751 2645 ts_tablet_manager.cc:1397] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 09:00:00.916234 2645 raft_consensus.cc:348] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.916345 2645 raft_consensus.cc:374] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:00.916378 2645 raft_consensus.cc:729] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97d732584249de9181b33bba844cfd, State: Initialized, Role: FOLLOWER
I20260620 09:00:00.917023 2694 ts_tablet_manager.cc:1905] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:00.917100 2645 consensus_queue.cc:260] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.917404 2645 ts_tablet_manager.cc:1428] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:00.922441 2695 ts_tablet_manager.cc:1918] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:00.922555 2695 log.cc:1196] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/4d08d8ac60754b4589dc58990c7d5b77
I20260620 09:00:00.922861 2695 ts_tablet_manager.cc:1939] T 4d08d8ac60754b4589dc58990c7d5b77 P cb0508912a224fe48abe51b735e32d8a: Deleting consensus metadata
I20260620 09:00:00.924144 2121 catalog_manager.cc:4841] TS cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941): tablet 4d08d8ac60754b4589dc58990c7d5b77 (table test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]) successfully deleted
I20260620 09:00:00.926038 2694 ts_tablet_manager.cc:1918] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:00.926155 2694 log.cc:1196] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/4d08d8ac60754b4589dc58990c7d5b77
I20260620 09:00:00.926468 2694 ts_tablet_manager.cc:1939] T 4d08d8ac60754b4589dc58990c7d5b77 P bd97d732584249de9181b33bba844cfd: Deleting consensus metadata
I20260620 09:00:00.927749 2121 catalog_manager.cc:4841] TS bd97d732584249de9181b33bba844cfd (127.0.98.1:37195): tablet 4d08d8ac60754b4589dc58990c7d5b77 (table test-table [id=f7e3d37e26f14dd0b0610a570a29a88f]) successfully deleted
I20260620 09:00:00.970192 2651 raft_consensus.cc:482] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:00.970355 2651 raft_consensus.cc:504] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.970813 2651 leader_election.cc:288] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:00.971290 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187449d10fd14507949363820ac7d67b" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a" is_pre_election: true
I20260620 09:00:00.971304 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187449d10fd14507949363820ac7d67b" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e" is_pre_election: true
I20260620 09:00:00.971455 2574 raft_consensus.cc:2454] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:00.971467 2434 raft_consensus.cc:2454] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:00.971781 2228 leader_election.cc:302] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, bd97d732584249de9181b33bba844cfd; no voters:
I20260620 09:00:00.971974 2651 raft_consensus.cc:2781] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:00.972041 2651 raft_consensus.cc:482] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:00.972081 2651 raft_consensus.cc:3037] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.973172 2651 raft_consensus.cc:504] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.973526 2651 leader_election.cc:288] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Requested vote from peers cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:00.973881 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187449d10fd14507949363820ac7d67b" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a"
I20260620 09:00:00.973886 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187449d10fd14507949363820ac7d67b" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e"
I20260620 09:00:00.974011 2574 raft_consensus.cc:3037] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.974015 2434 raft_consensus.cc:3037] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:00.974918 2434 raft_consensus.cc:2454] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:00.975158 2574 raft_consensus.cc:2454] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:00.975236 2225 leader_election.cc:302] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd97d732584249de9181b33bba844cfd, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:00.975400 2651 raft_consensus.cc:2781] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:00.975490 2651 raft_consensus.cc:686] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 1 LEADER]: Becoming Leader. State: Replica: bd97d732584249de9181b33bba844cfd, State: Running, Role: LEADER
I20260620 09:00:00.975617 2651 consensus_queue.cc:237] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:00.977366 2135 catalog_manager.cc:5495] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to bd97d732584249de9181b33bba844cfd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd97d732584249de9181b33bba844cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:01.011730 2573 raft_consensus.cc:1261] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:01.011775 2434 raft_consensus.cc:1261] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:01.012159 2651 consensus_queue.cc:1035] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:01.012328 2654 consensus_queue.cc:1035] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:00:01.029593 2673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:00:02.585551 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:02.585696 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:02.588218 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Sending DeleteTablet for 3 replicas of tablet 187449d10fd14507949363820ac7d67b
I20260620 09:00:02.588932 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 187449d10fd14507949363820ac7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58488
I20260620 09:00:02.588981 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 187449d10fd14507949363820ac7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42108
I20260620 09:00:02.588980 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 187449d10fd14507949363820ac7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47248
I20260620 09:00:02.589335 2728 tablet_replica.cc:330] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: stopping tablet replica
I20260620 09:00:02.589485 2728 raft_consensus.cc:2229] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:02.589622 2728 raft_consensus.cc:2258] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.589864 2729 tablet_replica.cc:330] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: stopping tablet replica
I20260620 09:00:02.590005 2729 raft_consensus.cc:2229] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:02.590119 2729 raft_consensus.cc:2258] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.590332 2728 ts_tablet_manager.cc:1905] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.590852 2729 ts_tablet_manager.cc:1905] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.592872 2727 tablet_replica.cc:330] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: stopping tablet replica
I20260620 09:00:02.593010 2727 raft_consensus.cc:2229] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:02.593248 2727 raft_consensus.cc:2258] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.593827 2727 ts_tablet_manager.cc:1905] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.593904 2728 ts_tablet_manager.cc:1918] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:00:02.594012 2728 log.cc:1196] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/187449d10fd14507949363820ac7d67b
I20260620 09:00:02.594208 2729 ts_tablet_manager.cc:1918] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:00:02.594283 2729 log.cc:1196] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/187449d10fd14507949363820ac7d67b
I20260620 09:00:02.594323 2728 ts_tablet_manager.cc:1939] T 187449d10fd14507949363820ac7d67b P cb0508912a224fe48abe51b735e32d8a: Deleting consensus metadata
I20260620 09:00:02.594584 2729 ts_tablet_manager.cc:1939] T 187449d10fd14507949363820ac7d67b P b2d3d58f0a0a435fb77814f962844c2e: Deleting consensus metadata
I20260620 09:00:02.595000 2121 catalog_manager.cc:4841] TS cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941): tablet 187449d10fd14507949363820ac7d67b (table test-table [id=4c2c51207e1f4a629b89fe574590c0ff]) successfully deleted
I20260620 09:00:02.595523 2124 catalog_manager.cc:4841] TS b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157): tablet 187449d10fd14507949363820ac7d67b (table test-table [id=4c2c51207e1f4a629b89fe574590c0ff]) successfully deleted
I20260620 09:00:02.596307 2727 ts_tablet_manager.cc:1918] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:00:02.596370 2727 log.cc:1196] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/187449d10fd14507949363820ac7d67b
I20260620 09:00:02.596692 2727 ts_tablet_manager.cc:1939] T 187449d10fd14507949363820ac7d67b P bd97d732584249de9181b33bba844cfd: Deleting consensus metadata
I20260620 09:00:02.597337 2121 catalog_manager.cc:4841] TS bd97d732584249de9181b33bba844cfd (127.0.98.1:37195): tablet 187449d10fd14507949363820ac7d67b (table test-table [id=4c2c51207e1f4a629b89fe574590c0ff]) successfully deleted
Jun 20 09:00:02 dist-test-slave-scsf krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946002, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:00:02 dist-test-slave-scsf krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946002, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:02.617240 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.608522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.0.98.62:33717)
0620 09:00:02.608787 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.608794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.608878 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:00:02.609001 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.609004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.609051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.609131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:02.609828 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.610369 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.611208 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.611423 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.611500 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.613997 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:02.614000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:02.614002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:02.614026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:02.615600 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:02.616020 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:02.616023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:02.616024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:02.616100 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:02.616331 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:02.616333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:02.616334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:02.616589 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:02.616692 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.616931 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.617059 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
I20260620 09:00:02.619671 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:02.620122 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:02.624732 2274 tablet_service.cc:1467] Processing CreateTablet for tablet d946ea787a004b038aa704bb23317827 (DEFAULT_TABLE table=test-table [id=310f9b7f74fd470a9e98416a9e460ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.624769 2554 tablet_service.cc:1467] Processing CreateTablet for tablet d946ea787a004b038aa704bb23317827 (DEFAULT_TABLE table=test-table [id=310f9b7f74fd470a9e98416a9e460ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.624758 2414 tablet_service.cc:1467] Processing CreateTablet for tablet d946ea787a004b038aa704bb23317827 (DEFAULT_TABLE table=test-table [id=310f9b7f74fd470a9e98416a9e460ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.625036 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946ea787a004b038aa704bb23317827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.625036 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946ea787a004b038aa704bb23317827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.625070 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946ea787a004b038aa704bb23317827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.627041 2738 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Bootstrap starting.
I20260620 09:00:02.627862 2740 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Bootstrap starting.
I20260620 09:00:02.627943 2738 tablet_bootstrap.cc:654] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.628053 2739 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Bootstrap starting.
I20260620 09:00:02.628842 2740 tablet_bootstrap.cc:654] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.628907 2739 tablet_bootstrap.cc:654] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.629140 2738 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: No bootstrap required, opened a new log
I20260620 09:00:02.629247 2738 ts_tablet_manager.cc:1397] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:02.629760 2738 raft_consensus.cc:348] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.629880 2738 raft_consensus.cc:374] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.629912 2738 raft_consensus.cc:729] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.630053 2740 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: No bootstrap required, opened a new log
I20260620 09:00:02.630060 2739 tablet_bootstrap.cc:492] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: No bootstrap required, opened a new log
I20260620 09:00:02.630064 2738 consensus_queue.cc:260] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.630137 2739 ts_tablet_manager.cc:1397] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:00:02.630137 2740 ts_tablet_manager.cc:1397] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:02.630522 2739 raft_consensus.cc:348] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.630513 2740 raft_consensus.cc:348] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.630625 2739 raft_consensus.cc:374] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.630625 2740 raft_consensus.cc:374] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.630658 2740 raft_consensus.cc:729] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97d732584249de9181b33bba844cfd, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.630658 2739 raft_consensus.cc:729] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.630798 2739 consensus_queue.cc:260] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.630827 2740 consensus_queue.cc:260] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.631100 2740 ts_tablet_manager.cc:1428] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:02.631171 2739 ts_tablet_manager.cc:1428] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:02.631745 2738 ts_tablet_manager.cc:1428] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:02.681684 2744 raft_consensus.cc:482] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:02.681851 2744 raft_consensus.cc:504] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.682971 2744 leader_election.cc:288] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd97d732584249de9181b33bba844cfd (127.0.98.1:37195), cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941)
I20260620 09:00:02.687670 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.683323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34703 (local address 127.0.98.1:37195)
0620 09:00:02.683632 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.683641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.683660 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:00:02.683751 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.683754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.683809 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.683915 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:02.684244 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.685045 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.686359 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.686952 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.687095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.687180 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.687449 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.687523 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
I20260620 09:00:02.687670 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.683749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35799 (local address 127.0.98.2:32941)
0620 09:00:02.684027 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.684036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.684050 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:02.684117 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.684119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.684168 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.684259 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:02.684534 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.685392 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.686489 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.687100 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.687201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.687255 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.687448 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.687561 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
I20260620 09:00:02.687703 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.683629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.98.3:35799)
0620 09:00:02.683935 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:02.683950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:02.684037 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:02.684270 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:02.684273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:02.684288 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:02.684403 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.684408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.685566 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.685570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.686374 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.686383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.687195 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.687198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.687340 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.687357 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:00:02.687417 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:00:02.687518 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":63,"thread_start_us":85,"threads_started":1}
I20260620 09:00:02.687711 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.683204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.98.3:34703)
0620 09:00:02.683506 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:02.683523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:02.683644 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:02.683962 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:02.683965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:02.683984 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:02.684116 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.684122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.685148 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.685155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.686237 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.686245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.687125 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.687127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.687243 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.687339 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:00:02.687419 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:00:02.687527 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":80,"thread_start_us":144,"threads_started":1}
I20260620 09:00:02.688410 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946ea787a004b038aa704bb23317827" candidate_uuid: "b2d3d58f0a0a435fb77814f962844c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd97d732584249de9181b33bba844cfd" is_pre_election: true
I20260620 09:00:02.688580 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946ea787a004b038aa704bb23317827" candidate_uuid: "b2d3d58f0a0a435fb77814f962844c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a" is_pre_election: true
I20260620 09:00:02.688639 2294 raft_consensus.cc:2454] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d3d58f0a0a435fb77814f962844c2e in term 0.
I20260620 09:00:02.688741 2434 raft_consensus.cc:2454] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d3d58f0a0a435fb77814f962844c2e in term 0.
I20260620 09:00:02.689015 2505 leader_election.cc:302] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, bd97d732584249de9181b33bba844cfd; no voters:
I20260620 09:00:02.689256 2744 raft_consensus.cc:2781] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:02.689316 2744 raft_consensus.cc:482] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:02.689352 2744 raft_consensus.cc:3037] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.690340 2744 raft_consensus.cc:504] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.690726 2744 leader_election.cc:288] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [CANDIDATE]: Term 1 election: Requested vote from peers bd97d732584249de9181b33bba844cfd (127.0.98.1:37195), cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941)
I20260620 09:00:02.691227 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946ea787a004b038aa704bb23317827" candidate_uuid: "b2d3d58f0a0a435fb77814f962844c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a"
I20260620 09:00:02.691262 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946ea787a004b038aa704bb23317827" candidate_uuid: "b2d3d58f0a0a435fb77814f962844c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd97d732584249de9181b33bba844cfd"
I20260620 09:00:02.691359 2434 raft_consensus.cc:3037] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.691380 2294 raft_consensus.cc:3037] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.692335 2294 raft_consensus.cc:2454] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d3d58f0a0a435fb77814f962844c2e in term 1.
I20260620 09:00:02.692585 2434 raft_consensus.cc:2454] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d3d58f0a0a435fb77814f962844c2e in term 1.
I20260620 09:00:02.692893 2505 leader_election.cc:302] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d3d58f0a0a435fb77814f962844c2e, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:02.693078 2744 raft_consensus.cc:2781] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:02.693377 2744 raft_consensus.cc:686] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 1 LEADER]: Becoming Leader. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Running, Role: LEADER
I20260620 09:00:02.693639 2744 consensus_queue.cc:237] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.695986 2136 catalog_manager.cc:5495] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b2d3d58f0a0a435fb77814f962844c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } health_report { overall_health: HEALTHY } } }
I20260620 09:00:02.720819 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.716521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.0.98.3:33157)
0620 09:00:02.716833 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.716842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.716857 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:02.716942 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.716945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.717008 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.717173 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:02.717429 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.718257 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.719076 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.719440 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.719536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.719773 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:02.720346 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:02.720480 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.720584 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.720639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":145,"threads_started":1}
I20260620 09:00:02.731446 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.727997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54647 (local address 127.0.98.2:32941)
0620 09:00:02.728121 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.728125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.728311 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:00:02.728399 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.728403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.728469 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.728550 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:02.728765 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.729527 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.730334 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.730672 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.730768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.731045 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:02.731134 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:02.731213 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.731291 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.731338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:02.731467 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.727884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32941 (local address 127.0.98.3:54647)
0620 09:00:02.728194 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:02.728207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:02.728324 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:02.728565 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:02.728567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:02.728573 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:02.728664 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.728669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.729638 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.729641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.730225 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.730232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.730809 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.730811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.730877 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.730919 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:02.731220 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:02.731224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:02.731271 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:00:02.731338 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260620 09:00:02.737694 2294 raft_consensus.cc:1261] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Refusing update from remote peer b2d3d58f0a0a435fb77814f962844c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:02.737754 2434 raft_consensus.cc:1261] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Refusing update from remote peer b2d3d58f0a0a435fb77814f962844c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:02.738153 2744 consensus_queue.cc:1035] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:02.738358 2749 consensus_queue.cc:1035] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:02.756138 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.752941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58131 (local address 127.0.98.3:33157)
0620 09:00:02.753282 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:02.753288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:02.753303 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:02.753363 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:02.753364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:02.753404 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:02.753462 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:02.753745 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.754240 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.755070 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.755323 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.755403 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.755681 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:02.755768 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:02.755825 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:02.755962 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 09:00:02.756020 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260620 09:00:02.756165 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:02.752892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33157 (local address 127.0.98.2:58131)
0620 09:00:02.753155 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:02.753169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:02.753281 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:02.753496 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:02.753500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:02.753511 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:02.753660 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.753667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.754338 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.754340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.754927 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:02.754935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:02.755427 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:02.755451 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:02.755529 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:02.755557 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:02.755854 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:02.755864 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 09:00:02.755927 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:00:02.755996 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":114,"threads_started":1}
I20260620 09:00:02.781059 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:02.781253 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:02.783581 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Sending DeleteTablet for 3 replicas of tablet d946ea787a004b038aa704bb23317827
I20260620 09:00:02.784206 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet d946ea787a004b038aa704bb23317827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58488
I20260620 09:00:02.784570 2727 tablet_replica.cc:330] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: stopping tablet replica
I20260620 09:00:02.784545 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet d946ea787a004b038aa704bb23317827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47248
I20260620 09:00:02.784708 2727 raft_consensus.cc:2229] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:02.784852 2727 raft_consensus.cc:2258] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.785068 2728 tablet_replica.cc:330] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: stopping tablet replica
I20260620 09:00:02.785195 2728 raft_consensus.cc:2229] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:02.785315 2728 raft_consensus.cc:2258] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.785297 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet d946ea787a004b038aa704bb23317827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42108
I20260620 09:00:02.785470 2729 tablet_replica.cc:330] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: stopping tablet replica
I20260620 09:00:02.785424 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:00:02.785567 2729 raft_consensus.cc:2229] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 1 LEADER]: Raft consensus shutting down.
W20260620 09:00:02.785823 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:02.785858 2729 raft_consensus.cc:2258] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:02.786697 2728 ts_tablet_manager.cc:1905] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.786811 2727 ts_tablet_manager.cc:1905] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.789363 2727 ts_tablet_manager.cc:1918] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:02.789366 2728 ts_tablet_manager.cc:1918] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:02.789462 2727 log.cc:1196] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/d946ea787a004b038aa704bb23317827
I20260620 09:00:02.789474 2728 log.cc:1196] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/d946ea787a004b038aa704bb23317827
I20260620 09:00:02.789741 2728 ts_tablet_manager.cc:1939] T d946ea787a004b038aa704bb23317827 P cb0508912a224fe48abe51b735e32d8a: Deleting consensus metadata
I20260620 09:00:02.789750 2727 ts_tablet_manager.cc:1939] T d946ea787a004b038aa704bb23317827 P bd97d732584249de9181b33bba844cfd: Deleting consensus metadata
I20260620 09:00:02.790484 2121 catalog_manager.cc:4841] TS bd97d732584249de9181b33bba844cfd (127.0.98.1:37195): tablet d946ea787a004b038aa704bb23317827 (table test-table [id=310f9b7f74fd470a9e98416a9e460ac3]) successfully deleted
I20260620 09:00:02.790670 2121 catalog_manager.cc:4841] TS cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941): tablet d946ea787a004b038aa704bb23317827 (table test-table [id=310f9b7f74fd470a9e98416a9e460ac3]) successfully deleted
I20260620 09:00:02.791304 2729 ts_tablet_manager.cc:1905] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:02.794004 2729 ts_tablet_manager.cc:1918] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:02.794099 2729 log.cc:1196] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/d946ea787a004b038aa704bb23317827
I20260620 09:00:02.794350 2729 ts_tablet_manager.cc:1939] T d946ea787a004b038aa704bb23317827 P b2d3d58f0a0a435fb77814f962844c2e: Deleting consensus metadata
I20260620 09:00:02.795058 2124 catalog_manager.cc:4841] TS b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157): tablet d946ea787a004b038aa704bb23317827 (table test-table [id=310f9b7f74fd470a9e98416a9e460ac3]) successfully deleted
I20260620 09:00:02.795167 2554 tablet_service.cc:1467] Processing CreateTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 (DEFAULT_TABLE table=test-table [id=aab7a92ee027457e94c0e59a04bee14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.795365 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaae74c5f2a45ffb287f8a6c8856129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.797029 2739 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Bootstrap starting.
I20260620 09:00:02.797266 2274 tablet_service.cc:1467] Processing CreateTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 (DEFAULT_TABLE table=test-table [id=aab7a92ee027457e94c0e59a04bee14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.797490 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaae74c5f2a45ffb287f8a6c8856129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.797972 2739 tablet_bootstrap.cc:654] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.799397 2740 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd: Bootstrap starting.
I20260620 09:00:02.799710 2739 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: No bootstrap required, opened a new log
I20260620 09:00:02.799786 2739 ts_tablet_manager.cc:1397] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 09:00:02.800107 2740 tablet_bootstrap.cc:654] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.800205 2739 raft_consensus.cc:348] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.800308 2739 raft_consensus.cc:374] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.800338 2739 raft_consensus.cc:729] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d3d58f0a0a435fb77814f962844c2e, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.800501 2739 consensus_queue.cc:260] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.800786 2739 ts_tablet_manager.cc:1428] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:02.800969 2414 tablet_service.cc:1467] Processing CreateTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 (DEFAULT_TABLE table=test-table [id=aab7a92ee027457e94c0e59a04bee14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:02.801184 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaae74c5f2a45ffb287f8a6c8856129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:02.802626 2740 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd: No bootstrap required, opened a new log
I20260620 09:00:02.802712 2740 ts_tablet_manager.cc:1397] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:00:02.803126 2740 raft_consensus.cc:348] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.803231 2740 raft_consensus.cc:374] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.803265 2740 raft_consensus.cc:729] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97d732584249de9181b33bba844cfd, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.803360 2738 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Bootstrap starting.
I20260620 09:00:02.803391 2740 consensus_queue.cc:260] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.803663 2740 ts_tablet_manager.cc:1428] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:02.804229 2738 tablet_bootstrap.cc:654] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:02.805336 2738 tablet_bootstrap.cc:492] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: No bootstrap required, opened a new log
I20260620 09:00:02.805454 2738 ts_tablet_manager.cc:1397] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:00:02.805809 2738 raft_consensus.cc:348] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.805958 2738 raft_consensus.cc:374] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:02.806017 2738 raft_consensus.cc:729] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0508912a224fe48abe51b735e32d8a, State: Initialized, Role: FOLLOWER
I20260620 09:00:02.806141 2738 consensus_queue.cc:260] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.806515 2738 ts_tablet_manager.cc:1428] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:02.953406 2718 raft_consensus.cc:482] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:02.953605 2718 raft_consensus.cc:504] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.954066 2718 leader_election.cc:288] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:02.954762 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaae74c5f2a45ffb287f8a6c8856129" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a" is_pre_election: true
I20260620 09:00:02.954756 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaae74c5f2a45ffb287f8a6c8856129" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e" is_pre_election: true
I20260620 09:00:02.954912 2434 raft_consensus.cc:2454] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:02.954912 2573 raft_consensus.cc:2454] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 0.
I20260620 09:00:02.955193 2225 leader_election.cc:302] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd97d732584249de9181b33bba844cfd, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:02.955395 2718 raft_consensus.cc:2781] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:02.955456 2718 raft_consensus.cc:482] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:02.955482 2718 raft_consensus.cc:3037] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.956622 2718 raft_consensus.cc:504] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.956964 2718 leader_election.cc:288] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Requested vote from peers cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941), b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157)
I20260620 09:00:02.957394 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaae74c5f2a45ffb287f8a6c8856129" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0508912a224fe48abe51b735e32d8a"
I20260620 09:00:02.957532 2434 raft_consensus.cc:3037] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.957468 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaae74c5f2a45ffb287f8a6c8856129" candidate_uuid: "bd97d732584249de9181b33bba844cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d3d58f0a0a435fb77814f962844c2e"
I20260620 09:00:02.957649 2573 raft_consensus.cc:3037] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:02.958541 2434 raft_consensus.cc:2454] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:02.958772 2573 raft_consensus.cc:2454] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd97d732584249de9181b33bba844cfd in term 1.
I20260620 09:00:02.958832 2225 leader_election.cc:302] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd97d732584249de9181b33bba844cfd, cb0508912a224fe48abe51b735e32d8a; no voters:
I20260620 09:00:02.959014 2718 raft_consensus.cc:2781] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:02.959111 2718 raft_consensus.cc:686] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [term 1 LEADER]: Becoming Leader. State: Replica: bd97d732584249de9181b33bba844cfd, State: Running, Role: LEADER
I20260620 09:00:02.959239 2718 consensus_queue.cc:237] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } }
I20260620 09:00:02.961097 2136 catalog_manager.cc:5495] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to bd97d732584249de9181b33bba844cfd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd97d732584249de9181b33bba844cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97d732584249de9181b33bba844cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:03.003652 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:03.000511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.1:37195)
0620 09:00:03.000617 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:03.000622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:03.000681 (+ 59us) server_negotiation.cc:407] Connection header received
0620 09:00:03.000839 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:03.000842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:03.000896 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:03.000994 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:03.001234 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:03.001747 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:03.002604 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:03.002833 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:03.002953 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:03.003213 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:03.003290 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:03.003370 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:03.003457 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:00:03.003510 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:00:03.010181 2573 raft_consensus.cc:1261] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:03.010237 2434 raft_consensus.cc:1261] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Refusing update from remote peer bd97d732584249de9181b33bba844cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:03.010605 2722 consensus_queue.cc:1035] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d3d58f0a0a435fb77814f962844c2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:03.010802 2718 consensus_queue.cc:1035] T deaae74c5f2a45ffb287f8a6c8856129 P bd97d732584249de9181b33bba844cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0508912a224fe48abe51b735e32d8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:00:03.023780 2710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:00:04.572597 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:04.572752 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:04.575389 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df224e0603634c8db9011e14b5a9976e: Sending DeleteTablet for 3 replicas of tablet deaae74c5f2a45ffb287f8a6c8856129
I20260620 09:00:04.576102 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58488
I20260620 09:00:04.576102 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47248
I20260620 09:00:04.576510 2793 tablet_replica.cc:330] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: stopping tablet replica
I20260620 09:00:04.576522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2209
I20260620 09:00:04.576498 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet deaae74c5f2a45ffb287f8a6c8856129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42108
I20260620 09:00:04.577447 2793 raft_consensus.cc:2229] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:04.577606 2793 raft_consensus.cc:2258] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:04.578343 2793 ts_tablet_manager.cc:1905] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:04.580596 2794 tablet_replica.cc:330] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: stopping tablet replica
I20260620 09:00:04.580830 2794 raft_consensus.cc:2229] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:04.581005 2794 raft_consensus.cc:2258] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:04.581660 2794 ts_tablet_manager.cc:1905] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:04.581722 2793 ts_tablet_manager.cc:1918] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:00:04.581805 2793 log.cc:1196] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/deaae74c5f2a45ffb287f8a6c8856129
I20260620 09:00:04.582091 2793 ts_tablet_manager.cc:1939] T deaae74c5f2a45ffb287f8a6c8856129 P cb0508912a224fe48abe51b735e32d8a: Deleting consensus metadata
I20260620 09:00:04.582942 2121 catalog_manager.cc:4841] TS cb0508912a224fe48abe51b735e32d8a (127.0.98.2:32941): tablet deaae74c5f2a45ffb287f8a6c8856129 (table test-table [id=aab7a92ee027457e94c0e59a04bee14b]) successfully deleted
I20260620 09:00:04.584041 2794 ts_tablet_manager.cc:1918] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:00:04.584151 2794 log.cc:1196] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/deaae74c5f2a45ffb287f8a6c8856129
I20260620 09:00:04.584491 2794 ts_tablet_manager.cc:1939] T deaae74c5f2a45ffb287f8a6c8856129 P b2d3d58f0a0a435fb77814f962844c2e: Deleting consensus metadata
I20260620 09:00:04.585177 2124 catalog_manager.cc:4841] TS b2d3d58f0a0a435fb77814f962844c2e (127.0.98.3:33157): tablet deaae74c5f2a45ffb287f8a6c8856129 (table test-table [id=aab7a92ee027457e94c0e59a04bee14b]) successfully deleted
W20260620 09:00:04.586246 2121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37195 (error 108)
I20260620 09:00:04.586479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2349
I20260620 09:00:04.587908 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:04.587490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.0.1:58520)
0620 09:00:04.587743 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:04.587823 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
W20260620 09:00:04.588135 2121 catalog_manager.cc:4584] TS bd97d732584249de9181b33bba844cfd (127.0.98.1:37195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deaae74c5f2a45ffb287f8a6c8856129: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37195: connect: Connection refused (error 111)
I20260620 09:00:04.593918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2489
I20260620 09:00:04.600674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2105
2026-06-20T09:00:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6218 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:04 dist-test-slave-scsf krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:04 dist-test-slave-scsf krb5kdc[2799](info): set up 2 sockets
Jun 20 09:00:04 dist-test-slave-scsf krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260620 09:00:05.632615 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:05 dist-test-slave-scsf krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:05.755683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:05.870183 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:05.870553 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:05.870642 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:05.874976 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:05.875093 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:05.875130 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:05.875171 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:05.875208 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:05.880939 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:05.882160 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:05.883260 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:05.892205 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:05.892251 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:05.892246 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:05.892333 2812 server_base.cc:1034] running on GCE node
I20260620 09:00:05.893188 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:05.894412 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:05.895612 2812 hybrid_clock.cc:648] HybridClock initialized: now 1781946005895601 us; error 36 us; skew 500 ppm
Jun 20 09:00:05 dist-test-slave-scsf krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:05.901861 2812 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:05.902959 2812 webserver.cc:460] Webserver started at http://127.0.98.62:37047/ using document root <none> and password file <none>
I20260620 09:00:05.903573 2812 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:05.903671 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:05.903908 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:05.906155 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "c3d765947c6143c98cc85b2554ea0737"
format_stamp: "Formatted at 2026-06-20 09:00:05 on dist-test-slave-scsf"
server_key: "b3c7341ccdb3641a9536dbe61e9958a1"
server_key_iv: "8dd79fd11f0dc25b4acc4dc8c89c150d"
server_key_version: "encryptionkey@0"
I20260620 09:00:05.906744 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3d765947c6143c98cc85b2554ea0737"
format_stamp: "Formatted at 2026-06-20 09:00:05 on dist-test-slave-scsf"
server_key: "b3c7341ccdb3641a9536dbe61e9958a1"
server_key_iv: "8dd79fd11f0dc25b4acc4dc8c89c150d"
server_key_version: "encryptionkey@0"
I20260620 09:00:05.911175 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:05.913949 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:05.915271 2812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:05.915422 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "c3d765947c6143c98cc85b2554ea0737"
format_stamp: "Formatted at 2026-06-20 09:00:05 on dist-test-slave-scsf"
server_key: "b3c7341ccdb3641a9536dbe61e9958a1"
server_key_iv: "8dd79fd11f0dc25b4acc4dc8c89c150d"
server_key_version: "encryptionkey@0"
I20260620 09:00:05.915553 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:05.930138 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:05.933512 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:05.933779 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:05.941407 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45993
I20260620 09:00:05.941433 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45993 every 8 connection(s)
I20260620 09:00:05.942514 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:05.945447 2880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:05.951262 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: Bootstrap starting.
I20260620 09:00:05.952421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2812
I20260620 09:00:05.952702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:05.952962 392 external_mini_cluster.cc:1428] Setting key 99ed1e36e7994e30bf1cf1cc34b3728b
I20260620 09:00:05.954331 2880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:05.955143 2880 log.cc:824] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:05.957532 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: No bootstrap required, opened a new log
Jun 20 09:00:05 dist-test-slave-scsf krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:05.962239 2880 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260620 09:00:05.962555 2880 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:05.962631 2880 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d765947c6143c98cc85b2554ea0737, State: Initialized, Role: FOLLOWER
I20260620 09:00:05.963160 2880 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260620 09:00:05.963333 2880 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:05.963410 2880 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:05.963542 2880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:05.964553 2880 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260620 09:00:05.964952 2880 leader_election.cc:302] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d765947c6143c98cc85b2554ea0737; no voters:
I20260620 09:00:05.965325 2880 leader_election.cc:288] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:05.965404 2885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:05.965611 2885 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [term 1 LEADER]: Becoming Leader. State: Replica: c3d765947c6143c98cc85b2554ea0737, State: Running, Role: LEADER
I20260620 09:00:05.966019 2885 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260620 09:00:05.966732 2880 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:05.967267 2887 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d765947c6143c98cc85b2554ea0737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260620 09:00:05.967398 2887 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:05.968151 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d765947c6143c98cc85b2554ea0737. Latest consensus state: current_term: 1 leader_uuid: "c3d765947c6143c98cc85b2554ea0737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d765947c6143c98cc85b2554ea0737" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260620 09:00:05.968302 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:05.968869 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:05.954159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.62:45993)
0620 09:00:05.954649 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:05.954683 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:05.954719 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:00:05.955434 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:05.955454 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:05.955909 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:05.956247 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:05.957190 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:05.958275 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:05.959472 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:05.959964 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:05.960089 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:05.963149 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:05.963155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:05.963175 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:05.963213 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:05.965283 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:05.965826 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:05.965835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:05.965844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:05.965986 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:05.966387 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:05.966392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:05.966394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:05.966916 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:05.967148 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:05.967389 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 09:00:05.967744 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
W20260620 09:00:05.971976 2900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:05.972066 2900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:05.972201 2889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:05.973014 2889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:05.978963 2889 catalog_manager.cc:1347] Generated new cluster ID: c1d4d890f2df4a2abf5646dfee1c14f4
I20260620 09:00:05.979055 2889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:05.989781 2889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:05.991128 2889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:05.997516 2889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d765947c6143c98cc85b2554ea0737: Generated new TSK 0
I20260620 09:00:05.998294 2889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:06.036499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:06.152839 2906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:06.153108 2906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:06.153153 2906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:06.157285 2906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:06.157358 2906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:06.157478 2906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:06.163079 2906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:06.164191 2906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:06.165290 2906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:06.175259 2911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:06.175313 2912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:06.175262 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:06.175609 2906 server_base.cc:1034] running on GCE node
I20260620 09:00:06.176030 2906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:06.176690 2906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:06.177865 2906 hybrid_clock.cc:648] HybridClock initialized: now 1781946006177842 us; error 32 us; skew 500 ppm
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:06.180722 2906 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:06.181660 2906 webserver.cc:460] Webserver started at http://127.0.98.1:38979/ using document root <none> and password file <none>
I20260620 09:00:06.182267 2906 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:06.182376 2906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:06.182617 2906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:06.185212 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "629a8abc9ceb43d08d9141da75d6f202"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "d2954148d2bbe916da255ea7634ab9d3"
server_key_iv: "8781986d88dec85c4acdc56aaa6abdb1"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.185814 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "629a8abc9ceb43d08d9141da75d6f202"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "d2954148d2bbe916da255ea7634ab9d3"
server_key_iv: "8781986d88dec85c4acdc56aaa6abdb1"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.189567 2906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:00:06.191969 2921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.193287 2906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:06.193425 2906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "629a8abc9ceb43d08d9141da75d6f202"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "d2954148d2bbe916da255ea7634ab9d3"
server_key_iv: "8781986d88dec85c4acdc56aaa6abdb1"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.193550 2906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:06.210106 2906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:06.213568 2906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:06.213770 2906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:06.214349 2906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:06.215189 2906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:06.215256 2906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.215327 2906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:06.215377 2906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.224800 2906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45795
I20260620 09:00:06.224820 3034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45795 every 8 connection(s)
I20260620 09:00:06.225792 2906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:06.233361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 2906
I20260620 09:00:06.233476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:06.233773 392 external_mini_cluster.cc:1428] Setting key f8bf6b62f891c33cf00f748d496093f9
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:06.241360 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.227653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37405 (local address 127.0.98.62:45993)
0620 09:00:06.227793 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:06.227797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:06.228687 (+ 890us) server_negotiation.cc:407] Connection header received
0620 09:00:06.230100 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:06.230103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:06.230158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:06.230265 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:06.231427 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.231958 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.233335 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.233701 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.233820 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.236942 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:06.236948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:06.236954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:06.236986 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:06.238806 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.239448 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.239454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.239458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.239565 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.239936 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.239940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.239942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.240225 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:06.240346 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:06.240918 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 09:00:06.241101 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:06.242023 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.228047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.1:37405)
0620 09:00:06.228539 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:06.228583 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:06.229893 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:06.230435 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:06.230442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:06.230798 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:06.231281 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.231293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.232104 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.232107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.233134 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.233147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.233999 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.234003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.234208 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.234859 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:06.234875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:06.236718 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:06.238940 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.238945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.238959 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.239280 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.239677 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.239679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.239680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.239830 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.240336 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:06.240341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:06.240654 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 09:00:06.240812 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 09:00:06.241098 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260620 09:00:06.243417 3035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260620 09:00:06.243675 3035 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:06.244262 3035 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260620 09:00:06.246007 2844 ts_manager.cc:194] Registered new tserver with Master: 629a8abc9ceb43d08d9141da75d6f202 (127.0.98.1:45795)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:06.247608 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:06.262965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:06.376832 3040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:06.377096 3040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:06.377139 3040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:06.381420 3040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:06.381485 3040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:06.381548 3040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:06.387072 3040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:06.388231 3040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:06.389341 3040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:06.399264 3046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:06.399257 3048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:06.399277 3045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:06.399534 3040 server_base.cc:1034] running on GCE node
I20260620 09:00:06.400081 3040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:06.400743 3040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:06.401928 3040 hybrid_clock.cc:648] HybridClock initialized: now 1781946006401902 us; error 38 us; skew 500 ppm
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:06.404727 3040 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:06.405565 3040 webserver.cc:460] Webserver started at http://127.0.98.2:42655/ using document root <none> and password file <none>
I20260620 09:00:06.406119 3040 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:06.406206 3040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:06.406448 3040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:06.408416 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e6ade8cc9d243a7bec59f66ce5a5c78"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "0de74de6544e24e111ad53f62fcd51ec"
server_key_iv: "2f5bd85f813829c67343bd664a46efe7"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.409020 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e6ade8cc9d243a7bec59f66ce5a5c78"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "0de74de6544e24e111ad53f62fcd51ec"
server_key_iv: "2f5bd85f813829c67343bd664a46efe7"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.412824 3040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:06.415269 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.416586 3040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:06.416716 3040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "3e6ade8cc9d243a7bec59f66ce5a5c78"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "0de74de6544e24e111ad53f62fcd51ec"
server_key_iv: "2f5bd85f813829c67343bd664a46efe7"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.416850 3040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:06.431279 3040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:06.434762 3040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:06.434978 3040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:06.435606 3040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:06.436452 3040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:06.436539 3040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.436604 3040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:06.436645 3040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.445196 3040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36267
I20260620 09:00:06.445264 3168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36267 every 8 connection(s)
I20260620 09:00:06.446128 3040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:06.449646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3040
I20260620 09:00:06.449739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:06.450031 392 external_mini_cluster.cc:1428] Setting key 27cd67cc7e640ecb3b8779dc05e77bc6
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:06.460440 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.448088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47829 (local address 127.0.98.62:45993)
0620 09:00:06.448250 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:06.448254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:06.449153 (+ 899us) server_negotiation.cc:407] Connection header received
0620 09:00:06.450703 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:06.450706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:06.450759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:06.450839 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:06.452114 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.452747 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.453917 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.454173 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.454288 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.456806 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:06.456809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:06.456811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:06.456834 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:06.458317 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.458875 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.458879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.458881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.458961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.459310 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.459315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.459317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.459497 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:06.459600 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:06.460138 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 09:00:06.460263 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:06.461467 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.448441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.2:47829)
0620 09:00:06.448976 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:06.449023 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:06.450460 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:06.451018 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:06.451024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:06.451469 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:06.451976 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.451989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.452869 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.452872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.453793 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.453801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.454325 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.454329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.454506 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.455136 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:06.455152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:06.456640 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:06.458467 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.458473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.458489 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.458788 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.459062 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.459065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.459067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.459209 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.459610 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:06.459619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:06.459899 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 09:00:06.460087 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 09:00:06.460481 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":103,"threads_started":1}
I20260620 09:00:06.462728 3169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260620 09:00:06.463027 3169 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:06.463521 3169 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260620 09:00:06.464565 2844 ts_manager.cc:194] Registered new tserver with Master: 3e6ade8cc9d243a7bec59f66ce5a5c78 (127.0.98.2:36267)
I20260620 09:00:06.465132 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:06.477141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:06.589464 3174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:06.589742 3174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:06.589787 3174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:06.593894 3174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:06.593955 3174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:06.594017 3174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:06.599967 3174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:06.601106 3174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:06.602061 3174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:06.611533 3180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:06.611621 3174 server_base.cc:1034] running on GCE node
W20260620 09:00:06.611527 3179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:06.611527 3182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:06.612325 3174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:06.613027 3174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:06.614199 3174 hybrid_clock.cc:648] HybridClock initialized: now 1781946006614182 us; error 29 us; skew 500 ppm
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:06.617050 3174 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:06.617967 3174 webserver.cc:460] Webserver started at http://127.0.98.3:45231/ using document root <none> and password file <none>
I20260620 09:00:06.618538 3174 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:06.618654 3174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:06.618882 3174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:06.621251 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e2cf0d4718d47f4ba57225e7025cce7"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "cd2802978622bf113b4b13044804fa61"
server_key_iv: "0ab2ef06e8745d187344705b9a835288"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.621763 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e2cf0d4718d47f4ba57225e7025cce7"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "cd2802978622bf113b4b13044804fa61"
server_key_iv: "0ab2ef06e8745d187344705b9a835288"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.625485 3174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:00:06.627803 3189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.629046 3174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:06.629164 3174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "6e2cf0d4718d47f4ba57225e7025cce7"
format_stamp: "Formatted at 2026-06-20 09:00:06 on dist-test-slave-scsf"
server_key: "cd2802978622bf113b4b13044804fa61"
server_key_iv: "0ab2ef06e8745d187344705b9a835288"
server_key_version: "encryptionkey@0"
I20260620 09:00:06.629247 3174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:06.654850 3174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:06.658432 3174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:06.658614 3174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:06.659168 3174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:06.659958 3174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:06.660003 3174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.660044 3174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:06.660063 3174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:06.669034 3174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43961
I20260620 09:00:06.669054 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43961 every 8 connection(s)
I20260620 09:00:06.669971 3174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:06.674044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3174
I20260620 09:00:06.674166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:06.674487 392 external_mini_cluster.cc:1428] Setting key e70228bdac08953b1161392e622ed04b
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:06.684031 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.671808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55777 (local address 127.0.98.62:45993)
0620 09:00:06.671952 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:06.671958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:06.672905 (+ 947us) server_negotiation.cc:407] Connection header received
0620 09:00:06.674517 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:06.674519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:06.674598 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:06.674688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:06.676232 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.676836 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.677992 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.678221 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.678331 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.680795 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:06.680798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:06.680801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:06.680824 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:06.682027 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.682539 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.682547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.682549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.682618 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:06.682962 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:06.682967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:06.682969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:06.683146 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:06.683253 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:06.683743 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 09:00:06.683871 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:00:06.684885 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:06.672283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.3:55777)
0620 09:00:06.672761 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:06.672802 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:06.674275 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:06.674893 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:06.674900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:06.675429 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:06.676053 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.676067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.676954 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.676958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.677871 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:06.677880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:06.678368 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:06.678372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:06.678546 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:06.679165 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:06.679189 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:06.680631 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:06.682145 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.682150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.682166 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.682425 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.682728 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:06.682731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:06.682733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:06.682867 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:06.683249 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:06.683256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:06.683505 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 09:00:06.683697 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 09:00:06.684023 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":171,"threads_started":1}
I20260620 09:00:06.686035 3303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260620 09:00:06.686273 3303 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:06.686806 3303 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260620 09:00:06.687920 2844 ts_manager.cc:194] Registered new tserver with Master: 6e2cf0d4718d47f4ba57225e7025cce7 (127.0.98.3:43961)
I20260620 09:00:06.688308 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:00:06.688668 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55777
W20260620 09:00:06.704484 2883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:06.703066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.98.62:45993)
0620 09:00:06.703249 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:06.703255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:06.703327 (+ 72us) server_negotiation.cc:407] Connection header received
0620 09:00:06.703441 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:06.703443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:06.703486 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:06.703556 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:06.704325 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49688: BlockingRecv error: recv got EOF from 127.0.0.1:49688 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 09:00:06.705410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2906
I20260620 09:00:06.711486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3040
I20260620 09:00:06.716800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3174
I20260620 09:00:06.721832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 2812
2026-06-20T09:00:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2119 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[3315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[3315](info): set up 2 sockets
Jun 20 09:00:06 dist-test-slave-scsf krb5kdc[3315](info): commencing operation
krb5kdc: starting...
W20260620 09:00:07.748494 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:07 dist-test-slave-scsf krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:07.869175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:07.978503 3328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:07.978806 3328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:07.978888 3328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:07.983222 3328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:07.983304 3328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:07.983342 3328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:07.983398 3328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:07.983438 3328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:07.988885 3328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:07.989972 3328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:07.991032 3328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:07.999449 3334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:07.999466 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:07.999482 3336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:07.999810 3328 server_base.cc:1034] running on GCE node
I20260620 09:00:08.000406 3328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:08.001559 3328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:08.002725 3328 hybrid_clock.cc:648] HybridClock initialized: now 1781946008002712 us; error 28 us; skew 500 ppm
Jun 20 09:00:08 dist-test-slave-scsf krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:08.005544 3328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:08.006424 3328 webserver.cc:460] Webserver started at http://127.0.98.62:43267/ using document root <none> and password file <none>
I20260620 09:00:08.006945 3328 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:08.007005 3328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:08.007169 3328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:08.009254 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "69192bbb58344ae7b82812f6d59dcbff"
format_stamp: "Formatted at 2026-06-20 09:00:08 on dist-test-slave-scsf"
server_key: "7d5a7b812ed4f6e971021efdabc051fe"
server_key_iv: "7a8bff41d43f31ef55cd320313f332d7"
server_key_version: "encryptionkey@0"
I20260620 09:00:08.009759 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "69192bbb58344ae7b82812f6d59dcbff"
format_stamp: "Formatted at 2026-06-20 09:00:08 on dist-test-slave-scsf"
server_key: "7d5a7b812ed4f6e971021efdabc051fe"
server_key_iv: "7a8bff41d43f31ef55cd320313f332d7"
server_key_version: "encryptionkey@0"
I20260620 09:00:08.013404 3328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:00:08.015831 3343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:08.017158 3328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:08.017298 3328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "69192bbb58344ae7b82812f6d59dcbff"
format_stamp: "Formatted at 2026-06-20 09:00:08 on dist-test-slave-scsf"
server_key: "7d5a7b812ed4f6e971021efdabc051fe"
server_key_iv: "7a8bff41d43f31ef55cd320313f332d7"
server_key_version: "encryptionkey@0"
I20260620 09:00:08.017408 3328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:08.034827 3328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:08.038064 3328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:08.038280 3328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:08.045543 3328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39633
I20260620 09:00:08.045802 3395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39633 every 8 connection(s)
I20260620 09:00:08.046505 3328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:08.049346 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:08.054961 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff: Bootstrap starting.
I20260620 09:00:08.055558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3328
I20260620 09:00:08.055689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:08.055958 392 external_mini_cluster.cc:1428] Setting key 577051ab04fedcc35b2834d781ea7bd4
I20260620 09:00:08.057883 3396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:08.058604 3396 log.cc:824] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:08.060797 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff: No bootstrap required, opened a new log
Jun 20 09:00:08 dist-test-slave-scsf krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:08.065410 3396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } }
I20260620 09:00:08.065696 3396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:08.065757 3396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69192bbb58344ae7b82812f6d59dcbff, State: Initialized, Role: FOLLOWER
I20260620 09:00:08.066278 3396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } }
I20260620 09:00:08.066463 3396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:08.066548 3396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:08.066694 3396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:08.067600 3396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } }
I20260620 09:00:08.067952 3396 leader_election.cc:302] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69192bbb58344ae7b82812f6d59dcbff; no voters:
I20260620 09:00:08.068315 3396 leader_election.cc:288] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:08.068392 3401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:08.068614 3401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [term 1 LEADER]: Becoming Leader. State: Replica: 69192bbb58344ae7b82812f6d59dcbff, State: Running, Role: LEADER
I20260620 09:00:08.068917 3401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } }
I20260620 09:00:08.069671 3396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:08.070546 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69192bbb58344ae7b82812f6d59dcbff. Latest consensus state: current_term: 1 leader_uuid: "69192bbb58344ae7b82812f6d59dcbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } } }
I20260620 09:00:08.070684 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:08.071656 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69192bbb58344ae7b82812f6d59dcbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69192bbb58344ae7b82812f6d59dcbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39633 } } }
I20260620 09:00:08.071750 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:08.071768 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:08.057283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.0.98.62:39633)
0620 09:00:08.057742 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:08.057756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:08.057790 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:00:08.058454 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:08.058471 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:08.058850 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:08.059198 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:08.060013 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:08.061113 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:08.062265 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:08.062719 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:08.062831 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:08.065696 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:08.065704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:08.065723 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:08.065756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:08.067589 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:08.068035 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:08.068041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:08.068047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:08.068199 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:08.068571 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:08.068573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:08.068575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:08.069112 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:08.069850 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:08.070017 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 09:00:08.070332 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":316,"thread_start_us":131,"threads_started":1}
I20260620 09:00:08.072383 3410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:08.074811 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:00:08.075498 3410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:08.081306 3410 catalog_manager.cc:1347] Generated new cluster ID: 234208bea22f41379687e8766b51843c
I20260620 09:00:08.081378 3410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:08.093196 3410 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:08.094470 3410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:08.097074 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:08.091080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.0.98.62:39633)
0620 09:00:08.091212 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:08.091218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:08.091491 (+ 273us) server_negotiation.cc:407] Connection header received
0620 09:00:08.091615 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:08.091618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:08.091678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:08.091805 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:08.092382 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:08.093190 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:08.094392 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:08.094780 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:08.094884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:08.095817 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:08.095821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:08.095824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:08.096403 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:08.096438 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:08.096457 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:08.096590 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:08.096694 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:00:08.096787 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:00:08.100311 3410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69192bbb58344ae7b82812f6d59dcbff: Generated new TSK 0
I20260620 09:00:08.101174 3410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:08.108768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3328
2026-06-20T09:00:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1384 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:08 dist-test-slave-scsf krb5kdc[3429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:08 dist-test-slave-scsf krb5kdc[3429](info): set up 2 sockets
Jun 20 09:00:08 dist-test-slave-scsf krb5kdc[3429](info): commencing operation
krb5kdc: starting...
W20260620 09:00:09.132373 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:09.252869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:09.362890 3442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:09.363232 3442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:09.363314 3442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:09.367987 3442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:09.368130 3442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:09.368175 3442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:09.368235 3442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:09.368269 3442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:09.373957 3442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:09.375145 3442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:09.376205 3442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:09.384783 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:09.384783 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:09.384857 3442 server_base.cc:1034] running on GCE node
W20260620 09:00:09.384806 3447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:09.385759 3442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:09.386809 3442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:09.387972 3442 hybrid_clock.cc:648] HybridClock initialized: now 1781946009387952 us; error 39 us; skew 500 ppm
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:09.390831 3442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:09.391695 3442 webserver.cc:460] Webserver started at http://127.0.98.62:36583/ using document root <none> and password file <none>
I20260620 09:00:09.392221 3442 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:09.392283 3442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:09.392518 3442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:09.394567 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "25782a53dbf94b818e93eedc3df2475a"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ffe274c34d9e0562faeae2753cae5877"
server_key_iv: "045fabe6ccd398a421b3c8ab0a147355"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.395079 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "25782a53dbf94b818e93eedc3df2475a"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ffe274c34d9e0562faeae2753cae5877"
server_key_iv: "045fabe6ccd398a421b3c8ab0a147355"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.399008 3442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:09.401789 3457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.403512 3442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:00:09.403666 3442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "25782a53dbf94b818e93eedc3df2475a"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ffe274c34d9e0562faeae2753cae5877"
server_key_iv: "045fabe6ccd398a421b3c8ab0a147355"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.403806 3442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:09.422005 3442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:09.425357 3442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:09.425516 3442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:09.432006 3442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38889
I20260620 09:00:09.432281 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38889 every 8 connection(s)
I20260620 09:00:09.433046 3442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:09.435732 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:09.439178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3442
I20260620 09:00:09.439294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:09.439591 392 external_mini_cluster.cc:1428] Setting key d5c85ee967b42f48d0c0c85f1684725d
I20260620 09:00:09.441843 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a: Bootstrap starting.
I20260620 09:00:09.444131 3510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:09.444962 3510 log.cc:824] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:09.446869 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a: No bootstrap required, opened a new log
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:09.450716 3510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 09:00:09.450973 3510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:09.451094 3510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25782a53dbf94b818e93eedc3df2475a, State: Initialized, Role: FOLLOWER
I20260620 09:00:09.451650 3510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 09:00:09.451828 3510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:09.451902 3510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:09.452011 3510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:09.453289 3510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 09:00:09.453684 3510 leader_election.cc:302] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25782a53dbf94b818e93eedc3df2475a; no voters:
I20260620 09:00:09.454099 3510 leader_election.cc:288] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:09.454216 3515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:09.454427 3515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [term 1 LEADER]: Becoming Leader. State: Replica: 25782a53dbf94b818e93eedc3df2475a, State: Running, Role: LEADER
I20260620 09:00:09.454494 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:09.441046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.98.62:38889)
0620 09:00:09.441512 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:09.441527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:09.441565 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:00:09.442243 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:09.442261 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:09.442602 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:09.442953 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:09.443864 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.444983 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.446053 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.446520 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.446695 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:09.449200 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:09.449205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:09.449218 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:09.449245 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:09.450935 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.451545 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.451550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.451554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.451666 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.452002 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.452006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.452008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.452581 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:09.452787 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:09.453031 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:00:09.453373 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260620 09:00:09.454784 3515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 09:00:09.455344 3510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:09.456588 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25782a53dbf94b818e93eedc3df2475a. Latest consensus state: current_term: 1 leader_uuid: "25782a53dbf94b818e93eedc3df2475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } } }
I20260620 09:00:09.456717 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:09.457126 3524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:09.457090 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25782a53dbf94b818e93eedc3df2475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25782a53dbf94b818e93eedc3df2475a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } } }
I20260620 09:00:09.457175 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:09.460068 3524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:09.465319 3524 catalog_manager.cc:1347] Generated new cluster ID: 5a3473f7ff7a4b90901b3843500f1b45
I20260620 09:00:09.465392 3524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:09.493772 3524 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:09.494753 3524 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:09.501956 3524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25782a53dbf94b818e93eedc3df2475a: Generated new TSK 0
I20260620 09:00:09.502903 3524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:09.538787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:09.651168 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:09.651446 3536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:09.651491 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:09.655555 3536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:09.655663 3536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:09.655761 3536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:09.661290 3536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:09.662320 3536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:09.663307 3536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:09.672792 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:09.672801 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:09.672801 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:09.673039 3536 server_base.cc:1034] running on GCE node
I20260620 09:00:09.673488 3536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:09.674121 3536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:09.675272 3536 hybrid_clock.cc:648] HybridClock initialized: now 1781946009675242 us; error 43 us; skew 500 ppm
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:09.678320 3536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:09.679188 3536 webserver.cc:460] Webserver started at http://127.0.98.1:44955/ using document root <none> and password file <none>
I20260620 09:00:09.679751 3536 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:09.679845 3536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:09.680073 3536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:09.682165 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "c535d925fb1d49e59dc20217024126c8"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ff0af4980941b5f0ed6b67fb7e54c149"
server_key_iv: "d988ce6184cf81a9148c7bcdbe334150"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.682765 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c535d925fb1d49e59dc20217024126c8"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ff0af4980941b5f0ed6b67fb7e54c149"
server_key_iv: "d988ce6184cf81a9148c7bcdbe334150"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.686602 3536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 09:00:09.689173 3551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.690430 3536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:09.690582 3536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "c535d925fb1d49e59dc20217024126c8"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ff0af4980941b5f0ed6b67fb7e54c149"
server_key_iv: "d988ce6184cf81a9148c7bcdbe334150"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.690708 3536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:09.714583 3536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:09.718340 3536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:09.718575 3536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:09.719267 3536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:09.720198 3536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:09.720274 3536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.720353 3536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:09.720409 3536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.730000 3536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44553
I20260620 09:00:09.730058 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44553 every 8 connection(s)
I20260620 09:00:09.731020 3536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:09.735318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3536
I20260620 09:00:09.735433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:09.735693 392 external_mini_cluster.cc:1428] Setting key d520deb2236b9fdac7414dd1547eeb63
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:09.746009 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:09.732965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37587 (local address 127.0.98.62:38889)
0620 09:00:09.733110 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:09.733114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:09.733999 (+ 885us) server_negotiation.cc:407] Connection header received
0620 09:00:09.735633 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:09.735636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:09.735699 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:09.735786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:09.737338 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.737885 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.739347 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.739577 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.739687 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:09.742298 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:09.742302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:09.742304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:09.742327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:09.743805 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.744313 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.744318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.744320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.744404 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.744804 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.744808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.744810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.744993 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:09.745108 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:09.745692 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 09:00:09.745826 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:00:09.746884 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:09.733371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38889 (local address 127.0.98.1:37587)
0620 09:00:09.733834 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:09.733879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:09.735332 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:09.736029 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:09.736041 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:09.736573 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:09.737180 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:09.737203 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.738019 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.738025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:09.739220 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:09.739230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.739724 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.739728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:09.739902 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:09.740589 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:09.740605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:09.742120 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:09.743927 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:09.743932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:09.743948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:09.744209 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:09.744544 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:09.744548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:09.744550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:09.744689 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:09.745106 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:09.745115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:09.745436 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 09:00:09.745637 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 09:00:09.745983 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:09.748204 3665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38889
I20260620 09:00:09.748502 3665 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:09.749001 3665 heartbeater.cc:511] Master 127.0.98.62:38889 requested a full tablet report, sending...
I20260620 09:00:09.750592 3474 ts_manager.cc:194] Registered new tserver with Master: c535d925fb1d49e59dc20217024126c8 (127.0.98.1:44553)
I20260620 09:00:09.751752 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:09.763861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:09.879072 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:09.879390 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:09.879467 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:09.883536 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:09.883641 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:09.883733 3670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:09.889150 3670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:09.890255 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:09.891331 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:09.901024 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:09.900969 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:09.901046 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:09.901139 3670 server_base.cc:1034] running on GCE node
I20260620 09:00:09.901674 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:09.902323 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:09.903537 3670 hybrid_clock.cc:648] HybridClock initialized: now 1781946009903520 us; error 29 us; skew 500 ppm
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:09.906309 3670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:09.907238 3670 webserver.cc:460] Webserver started at http://127.0.98.2:44943/ using document root <none> and password file <none>
I20260620 09:00:09.907806 3670 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:09.907896 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:09.908178 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:09.910250 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ce437a9a32547ea813039be785fd8ff"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ddf8788b4e4c7477e17c0e0b5fdcedf7"
server_key_iv: "a11f3ac92be9c8d5f3e8abf468ddccd9"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.910800 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ce437a9a32547ea813039be785fd8ff"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ddf8788b4e4c7477e17c0e0b5fdcedf7"
server_key_iv: "a11f3ac92be9c8d5f3e8abf468ddccd9"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.914577 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:09.917004 3685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.918174 3670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:09.918308 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "9ce437a9a32547ea813039be785fd8ff"
format_stamp: "Formatted at 2026-06-20 09:00:09 on dist-test-slave-scsf"
server_key: "ddf8788b4e4c7477e17c0e0b5fdcedf7"
server_key_iv: "a11f3ac92be9c8d5f3e8abf468ddccd9"
server_key_version: "encryptionkey@0"
I20260620 09:00:09.918421 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:09.932219 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:09.935151 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:09.935355 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:09.935994 3670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:09.936847 3670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:09.936892 3670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.936933 3670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:09.936949 3670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:09.945551 3670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36437
I20260620 09:00:09.945573 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36437 every 8 connection(s)
I20260620 09:00:09.946553 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:09.950059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3670
I20260620 09:00:09.950198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:09.950460 392 external_mini_cluster.cc:1428] Setting key f7d252a164665e5dcb56242175f6c7dd
Jun 20 09:00:09 dist-test-slave-scsf krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:09.961445 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:09.948395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37203 (local address 127.0.98.62:38889)
0620 09:00:09.948599 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:09.948602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:09.949384 (+ 782us) server_negotiation.cc:407] Connection header received
0620 09:00:09.950921 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:09.950924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:09.950985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:09.951091 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:09.952262 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.952848 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.954586 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.954811 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.954922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:09.957788 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:09.957792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:09.957795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:09.957829 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:09.959238 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.959762 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.959767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.959768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.959870 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:09.960220 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:09.960224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:09.960226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:09.960418 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:09.960578 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:09.961151 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 09:00:09.961276 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:09.962322 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:09.948796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38889 (local address 127.0.98.2:37203)
0620 09:00:09.949234 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:09.949277 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:09.950700 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:09.951267 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:09.951274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:09.951639 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:09.952124 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:09.952137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.952968 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.952973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:09.954460 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:09.954470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:09.954961 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:09.954965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:09.955144 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:09.955797 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:09.955821 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:09.957613 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:09.959359 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:09.959365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:09.959382 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:09.959661 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:09.959969 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:09.959972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:09.959974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:09.960125 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:09.960577 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:09.960584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:09.960883 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 09:00:09.961096 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 09:00:09.961428 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260620 09:00:09.963580 3799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38889
I20260620 09:00:09.963848 3799 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:09.964350 3799 heartbeater.cc:511] Master 127.0.98.62:38889 requested a full tablet report, sending...
I20260620 09:00:09.965502 3474 ts_manager.cc:194] Registered new tserver with Master: 9ce437a9a32547ea813039be785fd8ff (127.0.98.2:36437)
I20260620 09:00:09.966363 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:09.979115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:10.090660 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:10.090977 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:10.091059 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:10.095166 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:10.095252 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:10.095345 3804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:10.100906 3804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:10.102008 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:10.103068 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:00:10.112676 3804 server_base.cc:1034] running on GCE node
W20260620 09:00:10.112574 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:10.112592 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:10.112574 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:10.113288 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:10.113973 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:10.115159 3804 hybrid_clock.cc:648] HybridClock initialized: now 1781946010115136 us; error 34 us; skew 500 ppm
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:10.117815 3804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:10.118734 3804 webserver.cc:460] Webserver started at http://127.0.98.3:41365/ using document root <none> and password file <none>
I20260620 09:00:10.119300 3804 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:10.119392 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:10.119618 3804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:10.121789 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb4b302495d5421688798fb896f91206"
format_stamp: "Formatted at 2026-06-20 09:00:10 on dist-test-slave-scsf"
server_key: "cc2d3fa7274444bf6c1f309a63a24560"
server_key_iv: "255bdc7e12e1a5e4ea7dc9ff771c1628"
server_key_version: "encryptionkey@0"
I20260620 09:00:10.122357 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb4b302495d5421688798fb896f91206"
format_stamp: "Formatted at 2026-06-20 09:00:10 on dist-test-slave-scsf"
server_key: "cc2d3fa7274444bf6c1f309a63a24560"
server_key_iv: "255bdc7e12e1a5e4ea7dc9ff771c1628"
server_key_version: "encryptionkey@0"
I20260620 09:00:10.126085 3804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:00:10.128559 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:10.129746 3804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:00:10.129885 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "eb4b302495d5421688798fb896f91206"
format_stamp: "Formatted at 2026-06-20 09:00:10 on dist-test-slave-scsf"
server_key: "cc2d3fa7274444bf6c1f309a63a24560"
server_key_iv: "255bdc7e12e1a5e4ea7dc9ff771c1628"
server_key_version: "encryptionkey@0"
I20260620 09:00:10.130002 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:10.139109 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:10.142017 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:10.142233 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:10.142841 3804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:10.143651 3804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:10.143728 3804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:10.143797 3804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:10.143834 3804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:10.152840 3804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33857
I20260620 09:00:10.152861 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33857 every 8 connection(s)
I20260620 09:00:10.153757 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:10.155131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3804
I20260620 09:00:10.155251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:10.155561 392 external_mini_cluster.cc:1428] Setting key e607158d0d6e6e9546351ab049886f4a
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:10.167348 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:10.155760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47191 (local address 127.0.98.62:38889)
0620 09:00:10.155914 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:10.155918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:10.157096 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 09:00:10.158670 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:10.158673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:10.158716 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:10.158779 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:10.159845 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.160337 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.161511 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.161724 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.161803 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:10.164217 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:10.164221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:10.164222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:10.164243 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:10.165582 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:10.166062 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:10.166064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:10.166065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:10.166141 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:10.166422 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:10.166424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:10.166425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:10.166600 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:10.166673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:10.167131 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 09:00:10.167227 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 09:00:10.168226 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:10.156342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38889 (local address 127.0.98.3:47191)
0620 09:00:10.156920 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:10.156977 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:10.158480 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:10.158907 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:10.158912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:10.159267 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:10.159727 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:10.159740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.160480 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.160483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:10.161400 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:10.161409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.161848 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.161851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:10.161983 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:10.162659 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:10.162675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:10.164071 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:10.165693 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:10.165697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:10.165709 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:10.165955 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:10.166217 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:10.166219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:10.166220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:10.166348 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:10.166685 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:10.166689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:10.166909 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 09:00:10.167091 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 09:00:10.167337 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":155,"threads_started":1}
I20260620 09:00:10.169516 3933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38889
I20260620 09:00:10.169757 3933 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:10.170231 3933 heartbeater.cc:511] Master 127.0.98.62:38889 requested a full tablet report, sending...
I20260620 09:00:10.171149 3473 ts_manager.cc:194] Registered new tserver with Master: eb4b302495d5421688798fb896f91206 (127.0.98.3:33857)
I20260620 09:00:10.171669 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47191
I20260620 09:00:10.181628 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:10.206676 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:10.198128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.98.62:38889)
0620 09:00:10.198255 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:10.198261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:10.198435 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:00:10.198609 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:10.198611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:10.198656 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:10.198744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:10.199447 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.199980 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.200887 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:10.201131 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:10.201243 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:10.203607 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:10.203611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:10.203613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:10.203638 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:10.204945 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:10.205379 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:10.205383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:10.205385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:10.205465 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:10.205735 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:10.205740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:10.205742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:10.205984 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:10.206125 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:10.206363 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 09:00:10.206480 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 09:00:10.207130 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59098
I20260620 09:00:10.207949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3536
I20260620 09:00:10.213477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3670
I20260620 09:00:10.218706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3804
I20260620 09:00:10.223773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3442
2026-06-20T09:00:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2117 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3945](info): set up 2 sockets
Jun 20 09:00:10 dist-test-slave-scsf krb5kdc[3945](info): commencing operation
krb5kdc: starting...
W20260620 09:00:11.248566 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:11.371006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:11.480245 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:11.480598 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:11.480676 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:11.484725 3958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:11.484834 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:11.484869 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:11.484910 3958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:11.484939 3958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:11.490433 3958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:11.491585 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:11.492684 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:11.501055 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:11.501055 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:11.501170 3958 server_base.cc:1034] running on GCE node
W20260620 09:00:11.501077 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:11.501896 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:11.502985 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:11.504144 3958 hybrid_clock.cc:648] HybridClock initialized: now 1781946011504116 us; error 46 us; skew 500 ppm
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:11.513203 3958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:11.514137 3958 webserver.cc:460] Webserver started at http://127.0.98.62:45143/ using document root <none> and password file <none>
I20260620 09:00:11.514670 3958 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:11.514729 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:11.514895 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:11.517081 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "dffe9de95e1244e0babab6acf74c4ff3"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "476e2ea6fa3957735ac2e404101109dc"
server_key_iv: "4b3f43a53baa67065a249ae1eee1983a"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.517580 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "dffe9de95e1244e0babab6acf74c4ff3"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "476e2ea6fa3957735ac2e404101109dc"
server_key_iv: "4b3f43a53baa67065a249ae1eee1983a"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.521373 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:11.523753 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:11.525107 3958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:11.525218 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "dffe9de95e1244e0babab6acf74c4ff3"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "476e2ea6fa3957735ac2e404101109dc"
server_key_iv: "4b3f43a53baa67065a249ae1eee1983a"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.525310 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:11.539104 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:11.542578 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:11.542747 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:11.549953 3958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38865
I20260620 09:00:11.550020 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38865 every 8 connection(s)
I20260620 09:00:11.550946 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:11.553812 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:11.557384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 3958
I20260620 09:00:11.557502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:11.557804 392 external_mini_cluster.cc:1428] Setting key 6d44048cd0137d5970e8ce2e3a3b23f6
I20260620 09:00:11.559675 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: Bootstrap starting.
I20260620 09:00:11.562306 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:11.563061 4026 log.cc:824] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:11.565307 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: No bootstrap required, opened a new log
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:11.569175 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 09:00:11.569435 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:11.569520 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffe9de95e1244e0babab6acf74c4ff3, State: Initialized, Role: FOLLOWER
I20260620 09:00:11.570019 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 09:00:11.570171 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:11.570238 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:11.570343 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:11.571264 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 09:00:11.571693 4026 leader_election.cc:302] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dffe9de95e1244e0babab6acf74c4ff3; no voters:
I20260620 09:00:11.572033 4026 leader_election.cc:288] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:11.572152 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:11.572362 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [term 1 LEADER]: Becoming Leader. State: Replica: dffe9de95e1244e0babab6acf74c4ff3, State: Running, Role: LEADER
I20260620 09:00:11.572564 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:11.559252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.0.98.62:38865)
0620 09:00:11.559642 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:11.559663 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:11.559704 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:00:11.560407 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:11.560443 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:11.560850 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:11.561161 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:11.562089 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.563152 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.564218 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.564746 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.564867 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:11.567384 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:11.567389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:11.567402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:11.567427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:11.569242 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:11.569742 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:11.569747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:11.569751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:11.569861 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:11.570169 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:11.570172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:11.570174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:11.570712 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:11.570920 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:11.571224 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 09:00:11.571517 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":140,"threads_started":1}
I20260620 09:00:11.572755 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260620 09:00:11.573403 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:11.575425 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffe9de95e1244e0babab6acf74c4ff3. Latest consensus state: current_term: 1 leader_uuid: "dffe9de95e1244e0babab6acf74c4ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260620 09:00:11.575562 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:11.575786 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dffe9de95e1244e0babab6acf74c4ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe9de95e1244e0babab6acf74c4ff3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260620 09:00:11.576030 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3 [sys.catalog]: This master's current role is: LEADER
W20260620 09:00:11.578472 4045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:11.578557 4045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:11.578616 4046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:11.579499 4046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:11.585125 4046 catalog_manager.cc:1347] Generated new cluster ID: 4bbecdbc93b6478c811b341d40f5a636
I20260620 09:00:11.585201 4046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:11.602626 4046 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:11.603662 4046 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:11.610551 4046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dffe9de95e1244e0babab6acf74c4ff3: Generated new TSK 0
I20260620 09:00:11.611275 4046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:11.639695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:11.752373 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:11.752676 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:11.752720 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:11.756832 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:11.756901 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:11.756959 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:11.762280 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:11.763402 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:11.764439 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:11.773970 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:11.774289 4052 server_base.cc:1034] running on GCE node
W20260620 09:00:11.773939 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:11.773939 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:11.774938 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:11.775664 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:11.776827 4052 hybrid_clock.cc:648] HybridClock initialized: now 1781946011776795 us; error 45 us; skew 500 ppm
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:11.779523 4052 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:11.780597 4052 webserver.cc:460] Webserver started at http://127.0.98.1:41377/ using document root <none> and password file <none>
I20260620 09:00:11.781200 4052 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:11.781296 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:11.781535 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:11.783649 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "11c5d94ddf4f4f5bab9f35f4fa9f8f57"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "63509e72f8f82b3a306bd40ae156a61c"
server_key_iv: "39984dda2321015eea4c7dd6af145fa4"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.784266 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "11c5d94ddf4f4f5bab9f35f4fa9f8f57"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "63509e72f8f82b3a306bd40ae156a61c"
server_key_iv: "39984dda2321015eea4c7dd6af145fa4"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.787971 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:11.790877 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:11.792116 4052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:11.792258 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "11c5d94ddf4f4f5bab9f35f4fa9f8f57"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "63509e72f8f82b3a306bd40ae156a61c"
server_key_iv: "39984dda2321015eea4c7dd6af145fa4"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.792378 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:11.806351 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:11.809221 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:11.809425 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:11.810019 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:11.810868 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:11.810938 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:11.811013 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:11.811057 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:11.821277 4052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33117
I20260620 09:00:11.821302 4180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33117 every 8 connection(s)
I20260620 09:00:11.822254 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:11.826408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4052
I20260620 09:00:11.826526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:11.826839 392 external_mini_cluster.cc:1428] Setting key 497ab458d2d201101a41fe20cb7c8c36
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:11.837841 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:11.824171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56255 (local address 127.0.98.62:38865)
0620 09:00:11.824314 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:11.824317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:11.825231 (+ 914us) server_negotiation.cc:407] Connection header received
0620 09:00:11.826732 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:11.826735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:11.826795 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:11.826904 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:11.828439 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.829236 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.830862 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.831104 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.831194 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:11.834152 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:11.834155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:11.834157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:11.834182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:11.835651 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:11.836189 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:11.836194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:11.836196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:11.836305 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:11.836714 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:11.836719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:11.836721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:11.836911 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:11.837022 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:11.837540 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 09:00:11.837673 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:11.838745 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:11.824632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.1:56255)
0620 09:00:11.825082 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:11.825129 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:11.826421 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:11.827091 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:11.827098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:11.827573 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:11.828273 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:11.828293 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.829404 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.829408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:11.830712 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:11.830725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:11.831268 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:11.831276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:11.831508 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:11.832192 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:11.832219 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:11.833963 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:11.835792 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:11.835798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:11.835814 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:11.836084 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:11.836412 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:11.836415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:11.836417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:11.836617 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:11.837020 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:11.837028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:11.837294 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 09:00:11.837484 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 09:00:11.837833 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:11.840112 4181 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 09:00:11.840374 4181 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:11.840899 4181 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 09:00:11.842428 3990 ts_manager.cc:194] Registered new tserver with Master: 11c5d94ddf4f4f5bab9f35f4fa9f8f57 (127.0.98.1:33117)
I20260620 09:00:11.843565 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:11.855197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:11.965914 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:11.966197 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:11.966240 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:11.970254 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:11.970310 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:11.970367 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:11.976110 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:11.977242 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:11.978281 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:11.988292 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:11.988293 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:11.988292 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:11.988729 4186 server_base.cc:1034] running on GCE node
I20260620 09:00:11.989192 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:11.989886 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:11.991037 4186 hybrid_clock.cc:648] HybridClock initialized: now 1781946011991018 us; error 31 us; skew 500 ppm
Jun 20 09:00:11 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:11.993923 4186 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:11.994777 4186 webserver.cc:460] Webserver started at http://127.0.98.2:33429/ using document root <none> and password file <none>
I20260620 09:00:11.995345 4186 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:11.995435 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:11.995688 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:11.997937 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf3e080be792410581b0146c676769e6"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "83a3c00337a2951b819188a71371deff"
server_key_iv: "112a427f25f6114f39815b22f2b3727e"
server_key_version: "encryptionkey@0"
I20260620 09:00:11.998524 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf3e080be792410581b0146c676769e6"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "83a3c00337a2951b819188a71371deff"
server_key_iv: "112a427f25f6114f39815b22f2b3727e"
server_key_version: "encryptionkey@0"
I20260620 09:00:12.002250 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 09:00:12.004714 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.005946 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:00:12.006100 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "bf3e080be792410581b0146c676769e6"
format_stamp: "Formatted at 2026-06-20 09:00:11 on dist-test-slave-scsf"
server_key: "83a3c00337a2951b819188a71371deff"
server_key_iv: "112a427f25f6114f39815b22f2b3727e"
server_key_version: "encryptionkey@0"
I20260620 09:00:12.006217 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:12.017530 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:12.020900 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:12.021116 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:12.021698 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:12.022516 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:12.022594 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.022665 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:12.022720 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.031755 4186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36287
I20260620 09:00:12.031780 4314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36287 every 8 connection(s)
I20260620 09:00:12.032727 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:12.041823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4186
I20260620 09:00:12.041927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:12.042208 392 external_mini_cluster.cc:1428] Setting key a989ea291d88bf31abbba28d395bf4d5
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:12.047156 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.034538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37253 (local address 127.0.98.62:38865)
0620 09:00:12.034681 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:12.034685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:12.035533 (+ 848us) server_negotiation.cc:407] Connection header received
0620 09:00:12.036966 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:12.036970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:12.037029 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:12.037138 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:12.038294 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.038850 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.039992 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.040229 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.040343 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.042889 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:12.042892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:12.042895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:12.042938 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:12.044982 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.045612 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.045616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.045618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.045699 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.046072 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.046075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.046077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.046264 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:12.046367 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:12.046868 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 09:00:12.046995 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:00:12.048098 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.034945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.2:37253)
0620 09:00:12.035382 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:12.035423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:12.036752 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:12.037306 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:12.037313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:12.037682 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:12.038147 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:12.038160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.038975 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.038979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:12.039859 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:12.039868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.040382 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.040386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:12.040572 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.041131 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:12.041146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:12.042735 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:12.045127 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:12.045135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:12.045152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:12.045505 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:12.045794 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:12.045797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:12.045800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:12.045944 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:12.046377 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:12.046386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:12.046634 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 09:00:12.046827 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 09:00:12.047170 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
I20260620 09:00:12.049291 4315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 09:00:12.049551 4315 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:12.050084 4315 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 09:00:12.051153 3990 ts_manager.cc:194] Registered new tserver with Master: bf3e080be792410581b0146c676769e6 (127.0.98.2:36287)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:12.051775 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:12.069602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:12.180894 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:12.181200 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:12.181244 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:12.185477 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:12.185536 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:12.185601 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:12.190958 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:12.192155 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:12.193225 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:12.202843 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:12.202850 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:12.202904 4328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:12.203058 4320 server_base.cc:1034] running on GCE node
I20260620 09:00:12.203634 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:12.204206 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:12.205384 4320 hybrid_clock.cc:648] HybridClock initialized: now 1781946012205358 us; error 48 us; skew 500 ppm
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:12.208201 4320 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:12.209086 4320 webserver.cc:460] Webserver started at http://127.0.98.3:45575/ using document root <none> and password file <none>
I20260620 09:00:12.209672 4320 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:12.209759 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:12.209975 4320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:12.212015 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed93b68f81d5450190858737c72c83f8"
format_stamp: "Formatted at 2026-06-20 09:00:12 on dist-test-slave-scsf"
server_key: "0d475eee0c84c69cb1fb1dec03263b96"
server_key_iv: "bcfcffc35e4a5366abf9ac04c255aa77"
server_key_version: "encryptionkey@0"
I20260620 09:00:12.212638 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed93b68f81d5450190858737c72c83f8"
format_stamp: "Formatted at 2026-06-20 09:00:12 on dist-test-slave-scsf"
server_key: "0d475eee0c84c69cb1fb1dec03263b96"
server_key_iv: "bcfcffc35e4a5366abf9ac04c255aa77"
server_key_version: "encryptionkey@0"
I20260620 09:00:12.216537 4320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:00:12.219007 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.220268 4320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:00:12.220404 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "ed93b68f81d5450190858737c72c83f8"
format_stamp: "Formatted at 2026-06-20 09:00:12 on dist-test-slave-scsf"
server_key: "0d475eee0c84c69cb1fb1dec03263b96"
server_key_iv: "bcfcffc35e4a5366abf9ac04c255aa77"
server_key_version: "encryptionkey@0"
I20260620 09:00:12.220563 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:12.233238 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:12.236081 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:12.236267 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:12.236860 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:12.237624 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:12.237668 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.237706 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:12.237722 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:12.246531 4320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37013
I20260620 09:00:12.246594 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37013 every 8 connection(s)
I20260620 09:00:12.247467 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:12.256704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4320
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:12.256850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:12.257149 392 external_mini_cluster.cc:1428] Setting key 276d74c426aeecb69bd137c6290c11bc
I20260620 09:00:12.261214 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.249304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41559 (local address 127.0.98.62:38865)
0620 09:00:12.249470 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:12.249474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:12.250312 (+ 838us) server_negotiation.cc:407] Connection header received
0620 09:00:12.251692 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:12.251694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:12.251741 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:12.251816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:12.253002 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.253526 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.254607 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.254813 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.254886 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.257433 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:12.257437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:12.257443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:12.257477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:12.259022 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.259607 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.259617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.259619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.259692 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.260036 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.260041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.260043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.260236 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:12.260344 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:12.260910 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 09:00:12.261045 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:00:12.262202 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.249717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.3:41559)
0620 09:00:12.250154 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:12.250199 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:12.251478 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:12.251983 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:12.251990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:12.252353 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:12.252876 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:12.252890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.253638 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.253640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:12.254499 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:12.254508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.254941 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.254943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:12.255082 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.255616 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:12.255633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:12.257252 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:12.259144 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:12.259152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:12.259167 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:12.259499 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:12.259784 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:12.259789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:12.259792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:12.259941 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:12.260341 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:12.260348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:12.260678 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 09:00:12.260871 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 09:00:12.261217 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260620 09:00:12.263480 4449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260620 09:00:12.263780 4449 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:12.264324 4449 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260620 09:00:12.265359 3990 ts_manager.cc:194] Registered new tserver with Master: ed93b68f81d5450190858737c72c83f8 (127.0.98.3:37013)
I20260620 09:00:12.265925 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41559
I20260620 09:00:12.271175 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:00:12.296942 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.287331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.0.98.1:33117)
0620 09:00:12.287711 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:12.287731 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:12.287752 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:00:12.287799 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:12.287803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:12.287993 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:12.288165 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:12.288911 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.289682 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.290556 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.290929 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.291050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.293541 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:12.293549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:12.293563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:12.293588 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:12.295086 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.295513 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.295520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.295521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.295605 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.295916 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.295918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.295919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.296175 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:12.296347 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:12.296568 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 09:00:12.296760 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:12.307869 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:12.299840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.98.62:38865)
0620 09:00:12.299972 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:12.299977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:12.300046 (+ 69us) server_negotiation.cc:407] Connection header received
0620 09:00:12.300187 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:12.300189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:12.300230 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:12.300313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:12.300956 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.301474 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.302261 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:12.302482 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:12.302590 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:12.304968 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:12.304971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:12.304973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:12.304998 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:12.306175 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.306610 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.306613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.306614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.306689 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:12.306970 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:12.306974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:12.306976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:12.307221 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:12.307360 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:12.307593 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 09:00:12.307707 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 09:00:12.308312 3990 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764
I20260620 09:00:12.308975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4052
I20260620 09:00:12.314785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4186
I20260620 09:00:12.320560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4320
I20260620 09:00:12.325680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 3958
2026-06-20T09:00:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2102 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[4461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[4461](info): set up 2 sockets
Jun 20 09:00:12 dist-test-slave-scsf krb5kdc[4461](info): commencing operation
krb5kdc: starting...
W20260620 09:00:13.364699 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:13 dist-test-slave-scsf krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:13.485311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:13.595582 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:13.595856 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:13.595935 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:13.600000 4474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:13.600063 4474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:13.600082 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:13.600099 4474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:13.600116 4474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:13.605430 4474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:13.606472 4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:13.607481 4474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:13.615737 4482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:13.615708 4480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:13.615718 4479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:13.616067 4474 server_base.cc:1034] running on GCE node
I20260620 09:00:13.616670 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:13.617746 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:13.618909 4474 hybrid_clock.cc:648] HybridClock initialized: now 1781946013618893 us; error 31 us; skew 500 ppm
Jun 20 09:00:13 dist-test-slave-scsf krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:13.621889 4474 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:13.622634 4474 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:13.622700 4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:13.622923 4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:13.625072 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "66b99271221c406491be4f9ffb2318c0"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "9e988bb412129518b94de4dc0a24f097"
server_key_iv: "7cdcfef5170673b480fa3e2bd55893f6"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.625638 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "66b99271221c406491be4f9ffb2318c0"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "9e988bb412129518b94de4dc0a24f097"
server_key_iv: "7cdcfef5170673b480fa3e2bd55893f6"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.629482 4474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:13.631830 4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:13.633143 4474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:13.633285 4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "66b99271221c406491be4f9ffb2318c0"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "9e988bb412129518b94de4dc0a24f097"
server_key_iv: "7cdcfef5170673b480fa3e2bd55893f6"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.633400 4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:13.657521 4474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:13.660526 4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:13.660741 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:13.667758 4474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46739
I20260620 09:00:13.668021 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46739 every 8 connection(s)
I20260620 09:00:13.668504 4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:13.671277 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:13.671626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4474
I20260620 09:00:13.671758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:13.672066 392 external_mini_cluster.cc:1428] Setting key b4b2a19e3838bf329367cef6200edabd
I20260620 09:00:13.678609 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: Bootstrap starting.
I20260620 09:00:13.681071 4541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: Neither blocks nor log segments found. Creating new log.
Jun 20 09:00:13 dist-test-slave-scsf krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:13.681854 4541 log.cc:824] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:13.684271 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: No bootstrap required, opened a new log
I20260620 09:00:13.687728 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:13.673531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.98.62:46739)
0620 09:00:13.673975 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:13.673993 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:13.674037 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:00:13.674764 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:13.674787 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:13.675236 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:13.675621 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:13.676283 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.677832 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.678721 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.679363 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.679519 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:13.682063 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:13.682070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:13.682086 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:13.682116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:13.684121 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:13.684705 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:13.684711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:13.684717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:13.684858 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:13.685198 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:13.685201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:13.685203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:13.685733 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:13.685922 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:13.686134 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 09:00:13.686690 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260620 09:00:13.688982 4541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260620 09:00:13.689209 4541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:13.689280 4541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b99271221c406491be4f9ffb2318c0, State: Initialized, Role: FOLLOWER
I20260620 09:00:13.689765 4541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260620 09:00:13.689936 4541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:13.690011 4541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:13.690147 4541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:13.691143 4541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260620 09:00:13.691514 4541 leader_election.cc:302] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b99271221c406491be4f9ffb2318c0; no voters:
I20260620 09:00:13.691871 4541 leader_election.cc:288] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:13.692031 4546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:13.692270 4546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [term 1 LEADER]: Becoming Leader. State: Replica: 66b99271221c406491be4f9ffb2318c0, State: Running, Role: LEADER
I20260620 09:00:13.692668 4546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260620 09:00:13.693320 4541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:13.694474 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b99271221c406491be4f9ffb2318c0. Latest consensus state: current_term: 1 leader_uuid: "66b99271221c406491be4f9ffb2318c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } } }
I20260620 09:00:13.694618 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:13.694813 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b99271221c406491be4f9ffb2318c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b99271221c406491be4f9ffb2318c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } } }
I20260620 09:00:13.695168 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0 [sys.catalog]: This master's current role is: LEADER
W20260620 09:00:13.698340 4561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:13.698437 4561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:13.698509 4557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:13.699221 4557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:13.704349 4557 catalog_manager.cc:1347] Generated new cluster ID: 16e714ee224e453baa67c7a63e89b534
I20260620 09:00:13.704414 4557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:13.716393 4557 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:13.717532 4557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:13.731335 4557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: Generated new TSK 0
I20260620 09:00:13.732110 4557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:13.758924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:13.873075 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:13.873334 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:13.873379 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:13.877548 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:13.877606 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:13.877669 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:13.883306 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:13.884367 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:13.885434 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:13.894560 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:13.894766 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:13.894560 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:13.894995 4567 server_base.cc:1034] running on GCE node
I20260620 09:00:13.895459 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:13.896116 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:13.897300 4567 hybrid_clock.cc:648] HybridClock initialized: now 1781946013897284 us; error 28 us; skew 500 ppm
Jun 20 09:00:13 dist-test-slave-scsf krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:13.899983 4567 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:13.900765 4567 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:13.900853 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:13.901091 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:13.903093 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc63895aabe0437dac956d6e3d366da2"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "69f9dbadd37623778c923821518b7a79"
server_key_iv: "ac83e993d3fef66e667e99663c964187"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.903681 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc63895aabe0437dac956d6e3d366da2"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "69f9dbadd37623778c923821518b7a79"
server_key_iv: "ac83e993d3fef66e667e99663c964187"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.907330 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:00:13.909735 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:13.910956 4567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:13.911085 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "cc63895aabe0437dac956d6e3d366da2"
format_stamp: "Formatted at 2026-06-20 09:00:13 on dist-test-slave-scsf"
server_key: "69f9dbadd37623778c923821518b7a79"
server_key_iv: "ac83e993d3fef66e667e99663c964187"
server_key_version: "encryptionkey@0"
I20260620 09:00:13.911198 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:13.940411 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:13.943388 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:13.943595 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:13.944195 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:13.945068 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:13.945152 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:13.945227 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:13.945281 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:13.953948 4567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37013
I20260620 09:00:13.953969 4694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37013 every 8 connection(s)
I20260620 09:00:13.954566 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:13.955674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4567
I20260620 09:00:13.955801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:13.956146 392 external_mini_cluster.cc:1428] Setting key 43d3f187f95c095da6b8120b7ba15053
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 09:00:13 dist-test-slave-scsf krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:13.969602 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:13.956560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51727 (local address 127.0.98.62:46739)
0620 09:00:13.956721 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:13.956726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:13.957839 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 09:00:13.959820 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:13.959826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:13.959886 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:13.959994 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:13.961265 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.962211 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.963366 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.963761 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.963878 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:13.966239 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:13.966242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:13.966244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:13.966267 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:13.967528 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:13.968029 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:13.968032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:13.968034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:13.968107 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:13.968482 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:13.968485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:13.968487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:13.968680 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:13.968785 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:13.969279 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 09:00:13.969421 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:00:13.970443 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:13.957097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.1:51727)
0620 09:00:13.957668 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:13.957726 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:13.959603 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:13.960151 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:13.960157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:13.960595 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:13.961114 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:13.961129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.962351 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.962354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:13.963238 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:13.963247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:13.963903 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:13.963906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:13.964042 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:13.964695 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:13.964712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:13.966079 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:13.967639 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:13.967643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:13.967655 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:13.967926 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:13.968197 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:13.968200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:13.968202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:13.968352 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:13.968785 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:13.968792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:13.969031 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 09:00:13.969228 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 09:00:13.969567 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260620 09:00:13.971843 4695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260620 09:00:13.972151 4695 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:13.972756 4695 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260620 09:00:13.974407 4503 ts_manager.cc:194] Registered new tserver with Master: cc63895aabe0437dac956d6e3d366da2 (127.0.98.1:37013)
I20260620 09:00:13.975502 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:13.983678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:14.092726 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:14.092967 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:14.093055 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:14.097311 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:14.097370 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:14.097460 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:14.102823 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:14.103880 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:14.104972 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:14.114156 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:14.114156 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:14.114204 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:14.114462 4700 server_base.cc:1034] running on GCE node
I20260620 09:00:14.115036 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:14.115684 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:14.116839 4700 hybrid_clock.cc:648] HybridClock initialized: now 1781946014116825 us; error 25 us; skew 500 ppm
Jun 20 09:00:14 dist-test-slave-scsf krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:14.119614 4700 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:14.120361 4700 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:14.120478 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:14.120720 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:14.122768 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "5ccac8a90ab0f9531dacd90f22783929"
server_key_iv: "dcb042527dd9dade8cd6374f81cd326e"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.123347 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "5ccac8a90ab0f9531dacd90f22783929"
server_key_iv: "dcb042527dd9dade8cd6374f81cd326e"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.126995 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:00:14.129369 4714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.130622 4700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:14.130759 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "5ccac8a90ab0f9531dacd90f22783929"
server_key_iv: "dcb042527dd9dade8cd6374f81cd326e"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.130869 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:14.153772 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:14.156638 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:14.156844 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:14.157439 4700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:14.158246 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:14.158316 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.158385 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:14.158437 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.167004 4700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33751
I20260620 09:00:14.167021 4827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33751 every 8 connection(s)
I20260620 09:00:14.167629 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:14.169948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4700
I20260620 09:00:14.170079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:14.170390 392 external_mini_cluster.cc:1428] Setting key 76e0e283209ad3793786f32508521303
Jun 20 09:00:14 dist-test-slave-scsf krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:14.182111 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.169413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58991 (local address 127.0.98.62:46739)
0620 09:00:14.169578 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.169583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.170549 (+ 966us) server_negotiation.cc:407] Connection header received
0620 09:00:14.172321 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.172324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.172375 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.172471 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:14.173656 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.174204 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.175343 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.175641 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.175755 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.178209 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:14.178213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:14.178215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:14.178239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:14.179706 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.180325 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.180330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.180332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.180413 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.180893 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.180896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.180898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.181101 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:14.181214 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.181813 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.181942 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:00:14.182951 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.169862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.2:58991)
0620 09:00:14.170393 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.170455 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.172122 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.172656 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.172663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.173022 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:14.173509 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.173522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.174329 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.174333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.175219 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.175228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.175792 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.175797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.175985 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.176591 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:14.176613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:14.178030 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:14.179877 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:14.179883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:14.179899 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:14.180213 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:14.180555 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:14.180558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:14.180560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:14.180735 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:14.181224 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:14.181232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:14.181530 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.181759 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.182094 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":147,"threads_started":1}
I20260620 09:00:14.184301 4828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260620 09:00:14.184571 4828 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:14.185029 4828 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260620 09:00:14.186084 4503 ts_manager.cc:194] Registered new tserver with Master: bf0c63812c1a4ff1b57dd03d8cd88efc (127.0.98.2:33751)
I20260620 09:00:14.186803 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:14.197167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:14.305717 4833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:14.306042 4833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:14.306123 4833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:14.310206 4833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:14.310302 4833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:14.310396 4833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:14.315853 4833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:14.317019 4833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:14.318106 4833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:14.327414 4841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:14.327442 4833 server_base.cc:1034] running on GCE node
W20260620 09:00:14.327450 4838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:14.327622 4839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:14.328083 4833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:14.328755 4833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:14.329922 4833 hybrid_clock.cc:648] HybridClock initialized: now 1781946014329906 us; error 27 us; skew 500 ppm
Jun 20 09:00:14 dist-test-slave-scsf krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:14.332731 4833 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:14.333495 4833 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:14.333595 4833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:14.333912 4833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:14.336107 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "b207684e00464ad8a45557842da92c6c"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "2dbbbbb4b6fd3fcb9e0f0a26f4ed5036"
server_key_iv: "1ba014df0cf0338705f97ce181ba1a5d"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.336818 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b207684e00464ad8a45557842da92c6c"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "2dbbbbb4b6fd3fcb9e0f0a26f4ed5036"
server_key_iv: "1ba014df0cf0338705f97ce181ba1a5d"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.340674 4833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:14.343245 4847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.344583 4833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:14.344715 4833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "b207684e00464ad8a45557842da92c6c"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "2dbbbbb4b6fd3fcb9e0f0a26f4ed5036"
server_key_iv: "1ba014df0cf0338705f97ce181ba1a5d"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.344830 4833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:14.360307 4833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:14.363196 4833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:14.363408 4833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:14.363972 4833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:14.364832 4833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:14.364899 4833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.364960 4833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:14.365007 4833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.373755 4833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45889
I20260620 09:00:14.373771 4960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45889 every 8 connection(s)
I20260620 09:00:14.374378 4833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:14.383226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 4833
I20260620 09:00:14.383368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:14.383672 392 external_mini_cluster.cc:1428] Setting key 0791919e9cd715e1b425200cdec77a1c
Jun 20 09:00:14 dist-test-slave-scsf krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:14.388690 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.376320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36891 (local address 127.0.98.62:46739)
0620 09:00:14.376540 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.376544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.377554 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 09:00:14.378888 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.378891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.378949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.379077 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:14.380230 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.380860 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.382016 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.382238 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.382348 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.385281 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:14.385285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:14.385288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:14.385317 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:14.386897 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.387363 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.387365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.387367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.387436 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.387723 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.387725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.387727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.387886 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:14.387958 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.388420 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.388576 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:00:14.389564 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.376815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.3:36891)
0620 09:00:14.377311 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.377351 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.378672 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.379251 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.379258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.379618 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:14.380086 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.380099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.380982 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.380986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.381888 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.381897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.382385 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.382389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.382562 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.383172 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:14.383194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:14.385109 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:14.386998 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:14.387002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:14.387014 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:14.387278 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:14.387513 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:14.387515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:14.387516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:14.387646 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:14.387968 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:14.387972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:14.388222 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.388377 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.388694 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260620 09:00:14.390722 4961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260620 09:00:14.390925 4961 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:14.391326 4961 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260620 09:00:14.392292 4503 ts_manager.cc:194] Registered new tserver with Master: b207684e00464ad8a45557842da92c6c (127.0.98.3:45889)
I20260620 09:00:14.392887 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36891
I20260620 09:00:14.397855 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:00:14.406526 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.399759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.62:46739)
0620 09:00:14.399907 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.399912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.400109 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:00:14.400206 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.400208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.400247 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.400313 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:14.401070 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.401568 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.402358 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.402562 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.402649 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.403897 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:14.403900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:14.403902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:14.403923 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:14.405096 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.405488 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.405491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.405492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.405560 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:14.405788 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:14.405790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:14.405791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:14.405989 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:14.406088 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.406253 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.406338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:14.409338 4503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:14.412220 4503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:14.426970 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.420822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45889 (local address 127.0.0.1:58412)
0620 09:00:14.421495 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.421531 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.421696 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.422460 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.422468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.422490 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:14.422644 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.422654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.424109 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.424112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.425331 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.425341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426396 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.426520 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426561 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.426661 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.426794 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":3,"spinlock_wait_cycles":8960,"thread_start_us":90,"threads_started":1}
I20260620 09:00:14.427139 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.421695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.98.1:37013)
0620 09:00:14.422068 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.422075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.422095 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:00:14.422172 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.422179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.422346 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.422479 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:14.422802 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.423723 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.425470 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426272 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426377 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426514 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.426880 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.426973 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":100,"threads_started":1}
I20260620 09:00:14.427299 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.421844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.98.2:33751)
0620 09:00:14.422822 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.422834 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.422856 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:00:14.422927 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.422931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.423118 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.423268 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:14.423507 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.424404 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.425855 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426436 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426541 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426671 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.427020 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.427135 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":116,"threads_started":1}
I20260620 09:00:14.427660 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.420969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.0.98.3:45889)
0620 09:00:14.421328 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.421344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.421848 (+ 504us) server_negotiation.cc:407] Connection header received
0620 09:00:14.421938 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.421945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.422165 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.422313 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:14.422793 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.423966 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.425470 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426273 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426377 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426514 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.427397 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.427490 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":165,"threads_started":1}
I20260620 09:00:14.427198 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.420969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.0.1:33556)
0620 09:00:14.421932 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.421949 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.422038 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.422510 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.422514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.422544 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:14.422669 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.422678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.423840 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.423846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.425338 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.425343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426576 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.426714 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426755 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.426833 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.426939 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":70,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260620 09:00:14.427347 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.421628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33751 (local address 127.0.0.1:34208)
0620 09:00:14.422237 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.422250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.422334 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.423263 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.423266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.423284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:14.423372 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.423378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.424668 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.424671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.425736 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.425744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.426577 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.426579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.426717 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.426778 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.426841 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.426937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":84,"thread_start_us":104,"threads_started":1}
I20260620 09:00:14.430625 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 595691311d6645e384a8183d0c9cce58 (DEFAULT_TABLE table=test-table [id=fe4577e60bbf4098aceb3003bdc55ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:14.430701 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 595691311d6645e384a8183d0c9cce58 (DEFAULT_TABLE table=test-table [id=fe4577e60bbf4098aceb3003bdc55ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:14.431640 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595691311d6645e384a8183d0c9cce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:14.431720 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595691311d6645e384a8183d0c9cce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:14.431389 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 595691311d6645e384a8183d0c9cce58 (DEFAULT_TABLE table=test-table [id=fe4577e60bbf4098aceb3003bdc55ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:14.432291 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595691311d6645e384a8183d0c9cce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:14.438091 4978 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Bootstrap starting.
I20260620 09:00:14.439646 4980 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Bootstrap starting.
I20260620 09:00:14.440354 4978 tablet_bootstrap.cc:654] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:14.441421 4980 tablet_bootstrap.cc:654] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:14.441463 4978 log.cc:824] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:14.442648 4980 log.cc:824] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:14.443322 4979 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: Bootstrap starting.
I20260620 09:00:14.443585 4978 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: No bootstrap required, opened a new log
I20260620 09:00:14.443774 4978 ts_tablet_manager.cc:1397] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:00:14.444621 4980 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: No bootstrap required, opened a new log
I20260620 09:00:14.444799 4980 ts_tablet_manager.cc:1397] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:00:14.445261 4979 tablet_bootstrap.cc:654] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:14.446197 4979 log.cc:824] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:14.447962 4979 tablet_bootstrap.cc:492] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: No bootstrap required, opened a new log
I20260620 09:00:14.447739 4978 raft_consensus.cc:348] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.448042 4978 raft_consensus.cc:374] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:14.448143 4979 ts_tablet_manager.cc:1397] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:00:14.448150 4978 raft_consensus.cc:729] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b207684e00464ad8a45557842da92c6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:14.448660 4978 consensus_queue.cc:260] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.449139 4980 raft_consensus.cc:348] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.449376 4980 raft_consensus.cc:374] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:14.449460 4980 raft_consensus.cc:729] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc63895aabe0437dac956d6e3d366da2, State: Initialized, Role: FOLLOWER
I20260620 09:00:14.449682 4978 ts_tablet_manager.cc:1428] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:00:14.449901 4980 consensus_queue.cc:260] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.450085 4961 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
I20260620 09:00:14.450629 4695 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
I20260620 09:00:14.450986 4980 ts_tablet_manager.cc:1428] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:00:14.451570 4979 raft_consensus.cc:348] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.451846 4979 raft_consensus.cc:374] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:14.451930 4979 raft_consensus.cc:729] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0c63812c1a4ff1b57dd03d8cd88efc, State: Initialized, Role: FOLLOWER
I20260620 09:00:14.452374 4979 consensus_queue.cc:260] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.452986 4828 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
I20260620 09:00:14.453325 4979 ts_tablet_manager.cc:1428] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 09:00:14.455530 4696 tablet.cc:2369] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:14.504927 4984 raft_consensus.cc:482] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:14.505167 4984 raft_consensus.cc:504] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.506361 4984 leader_election.cc:288] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc63895aabe0437dac956d6e3d366da2 (127.0.98.1:37013), bf0c63812c1a4ff1b57dd03d8cd88efc (127.0.98.2:33751)
I20260620 09:00:14.511240 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.506723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33289 (local address 127.0.98.2:33751)
0620 09:00:14.506956 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.506961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.507326 (+ 365us) server_negotiation.cc:407] Connection header received
0620 09:00:14.507580 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.507584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.507645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.507700 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:14.508152 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.508890 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510088 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.510640 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510758 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.510823 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.511018 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.511107 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 09:00:14.511382 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.506615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.98.3:54885)
0620 09:00:14.507026 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.507039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.507177 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.507586 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.507589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.507607 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:14.507756 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.507767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.508839 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.508844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.510221 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.510234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.510931 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.511040 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.511069 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.511143 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.511224 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":31}
I20260620 09:00:14.511412 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.506798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54885 (local address 127.0.98.1:37013)
0620 09:00:14.506922 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.506926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.507272 (+ 346us) server_negotiation.cc:407] Connection header received
0620 09:00:14.507338 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.507341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.507397 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.507462 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:14.507897 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.508717 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510372 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.510809 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510933 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.510991 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.511207 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.511291 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:00:14.511838 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.506615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33751 (local address 127.0.98.3:33289)
0620 09:00:14.507230 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.507245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:14.507426 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:14.507854 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:14.507860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:14.507881 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:14.508020 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.508028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.509011 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.509014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.509956 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:14.509965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.510761 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.510764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:14.510861 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.510902 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:00:14.511628 (+ 726us) client_negotiation.cc:240] Negotiation successful
0620 09:00:14.511700 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":185,"thread_start_us":128,"threads_started":1}
I20260620 09:00:14.512086 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595691311d6645e384a8183d0c9cce58" candidate_uuid: "b207684e00464ad8a45557842da92c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc63895aabe0437dac956d6e3d366da2" is_pre_election: true
I20260620 09:00:14.512439 4649 raft_consensus.cc:2454] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b207684e00464ad8a45557842da92c6c in term 0.
I20260620 09:00:14.512611 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595691311d6645e384a8183d0c9cce58" candidate_uuid: "b207684e00464ad8a45557842da92c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" is_pre_election: true
I20260620 09:00:14.512910 4782 raft_consensus.cc:2454] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b207684e00464ad8a45557842da92c6c in term 0.
I20260620 09:00:14.513024 4848 leader_election.cc:302] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b207684e00464ad8a45557842da92c6c, cc63895aabe0437dac956d6e3d366da2; no voters:
I20260620 09:00:14.513397 4984 raft_consensus.cc:2781] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:14.513486 4984 raft_consensus.cc:482] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:14.513540 4984 raft_consensus.cc:3037] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:14.515013 4984 raft_consensus.cc:504] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.515398 4984 leader_election.cc:288] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [CANDIDATE]: Term 1 election: Requested vote from peers cc63895aabe0437dac956d6e3d366da2 (127.0.98.1:37013), bf0c63812c1a4ff1b57dd03d8cd88efc (127.0.98.2:33751)
I20260620 09:00:14.515769 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595691311d6645e384a8183d0c9cce58" candidate_uuid: "b207684e00464ad8a45557842da92c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc63895aabe0437dac956d6e3d366da2"
I20260620 09:00:14.515898 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595691311d6645e384a8183d0c9cce58" candidate_uuid: "b207684e00464ad8a45557842da92c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc"
I20260620 09:00:14.515929 4649 raft_consensus.cc:3037] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:14.516019 4782 raft_consensus.cc:3037] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:14.517203 4782 raft_consensus.cc:2454] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b207684e00464ad8a45557842da92c6c in term 1.
I20260620 09:00:14.517203 4649 raft_consensus.cc:2454] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b207684e00464ad8a45557842da92c6c in term 1.
I20260620 09:00:14.517493 4849 leader_election.cc:302] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b207684e00464ad8a45557842da92c6c, bf0c63812c1a4ff1b57dd03d8cd88efc; no voters:
I20260620 09:00:14.517681 4984 raft_consensus.cc:2781] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:14.517912 4984 raft_consensus.cc:686] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 1 LEADER]: Becoming Leader. State: Replica: b207684e00464ad8a45557842da92c6c, State: Running, Role: LEADER
I20260620 09:00:14.518252 4984 consensus_queue.cc:237] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } }
I20260620 09:00:14.521278 4504 catalog_manager.cc:5495] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to b207684e00464ad8a45557842da92c6c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b207684e00464ad8a45557842da92c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b207684e00464ad8a45557842da92c6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:14.555298 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.551624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.0.98.3:45889)
0620 09:00:14.551752 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.551758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.551796 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:00:14.552015 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.552018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.552082 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.552198 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:14.552465 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.552975 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.553825 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:14.554044 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:14.554152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:14.554454 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:14.554877 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:14.555016 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.555080 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.555137 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:00:14.561453 4782 raft_consensus.cc:1261] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 1 FOLLOWER]: Refusing update from remote peer b207684e00464ad8a45557842da92c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:14.561453 4649 raft_consensus.cc:1261] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2 [term 1 FOLLOWER]: Refusing update from remote peer b207684e00464ad8a45557842da92c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:14.562208 4988 consensus_queue.cc:1035] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0c63812c1a4ff1b57dd03d8cd88efc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:14.562409 4984 consensus_queue.cc:1035] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc63895aabe0437dac956d6e3d366da2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:14.578749 4504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:14.578949 4504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:14.581552 4504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66b99271221c406491be4f9ffb2318c0: Sending DeleteTablet for 3 replicas of tablet 595691311d6645e384a8183d0c9cce58
I20260620 09:00:14.582425 4895 tablet_service.cc:1514] Processing DeleteTablet for tablet 595691311d6645e384a8183d0c9cce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58412
I20260620 09:00:14.582499 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 595691311d6645e384a8183d0c9cce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33556
I20260620 09:00:14.582796 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 595691311d6645e384a8183d0c9cce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34208
I20260620 09:00:14.583266 5002 tablet_replica.cc:330] T 595691311d6645e384a8183d0c9cce58 P cc63895aabe0437dac956d6e3d366da2: stopping tablet replica
I20260620 09:00:14.583403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4567
I20260620 09:00:14.589799 5001 tablet_replica.cc:330] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: stopping tablet replica
I20260620 09:00:14.590142 5001 raft_consensus.cc:2229] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:14.590469 5003 tablet_replica.cc:330] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc: stopping tablet replica
W20260620 09:00:14.590435 4489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37013 (error 108)
I20260620 09:00:14.590507 5001 raft_consensus.cc:2258] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:14.590909 5003 raft_consensus.cc:2229] T 595691311d6645e384a8183d0c9cce58 P bf0c63812c1a4ff1b57dd03d8cd88efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:14.591249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4700
I20260620 09:00:14.592538 5001 ts_tablet_manager.cc:1905] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:14.594269 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.593796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.0.1:33560)
0620 09:00:14.594068 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.594143 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260620 09:00:14.594547 4489 catalog_manager.cc:4584] TS cc63895aabe0437dac956d6e3d366da2 (127.0.98.1:37013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 595691311d6645e384a8183d0c9cce58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37013: connect: Connection refused (error 111)
I20260620 09:00:14.595407 5001 ts_tablet_manager.cc:1918] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:14.595494 5001 log.cc:1196] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/595691311d6645e384a8183d0c9cce58
I20260620 09:00:14.595845 5001 ts_tablet_manager.cc:1939] T 595691311d6645e384a8183d0c9cce58 P b207684e00464ad8a45557842da92c6c: Deleting consensus metadata
I20260620 09:00:14.597106 4492 catalog_manager.cc:4841] TS b207684e00464ad8a45557842da92c6c (127.0.98.3:45889): tablet 595691311d6645e384a8183d0c9cce58 (table test-table [id=fe4577e60bbf4098aceb3003bdc55ca7]) successfully deleted
I20260620 09:00:14.598392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4833
I20260620 09:00:14.598560 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.598262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33751 (local address 127.0.0.1:34224)
0620 09:00:14.598408 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:14.598471 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:00:14.604003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 4474
2026-06-20T09:00:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2278 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T09:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:14Z Disabled control of system clock
I20260620 09:00:14.633847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39947
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:00:14.745208 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:14.745471 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:14.745509 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:14.749449 5008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:14.749504 5008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:14.749519 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:14.749536 5008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:14.749552 5008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:14.754257 5008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:14.755247 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:14.756594 5008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:14.764829 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:14.764829 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:14.764905 5008 server_base.cc:1034] running on GCE node
W20260620 09:00:14.764851 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:14.765610 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:14.766682 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:14.767875 5008 hybrid_clock.cc:648] HybridClock initialized: now 1781946014767849 us; error 50 us; skew 500 ppm
I20260620 09:00:14.769793 5008 webserver.cc:460] Webserver started at http://127.0.98.62:35421/ using document root <none> and password file <none>
I20260620 09:00:14.770399 5008 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:14.770493 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:14.770731 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:14.772819 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "27e19109f4ad4e3c81896576fa17d178"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "826d09ccde97b2ed2fb5f84a1624e5c0"
server_key_iv: "617cf4fe07a2e022a1ebefa95357b7e2"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.773386 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "27e19109f4ad4e3c81896576fa17d178"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "826d09ccde97b2ed2fb5f84a1624e5c0"
server_key_iv: "617cf4fe07a2e022a1ebefa95357b7e2"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.777073 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:00:14.779486 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:14.780794 5008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:14.780942 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "27e19109f4ad4e3c81896576fa17d178"
format_stamp: "Formatted at 2026-06-20 09:00:14 on dist-test-slave-scsf"
server_key: "826d09ccde97b2ed2fb5f84a1624e5c0"
server_key_iv: "617cf4fe07a2e022a1ebefa95357b7e2"
server_key_version: "encryptionkey@0"
I20260620 09:00:14.781086 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:14.785740 5008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:14.786358 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:14.786547 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:14.793009 5008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39947
I20260620 09:00:14.793011 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39947 every 8 connection(s)
I20260620 09:00:14.793931 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:14.797156 5075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:14.799371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5008
I20260620 09:00:14.799506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:14.799772 392 external_mini_cluster.cc:1428] Setting key a84723e6f4bd98c7059fd2603c0ecfea
I20260620 09:00:14.803264 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: Bootstrap starting.
I20260620 09:00:14.805940 5075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:14.805985 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:14.801400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.0.98.62:39947)
0620 09:00:14.802112 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:14.802132 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:14.802176 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:00:14.802964 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:14.802985 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:14.803436 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:14.803784 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:14.804535 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:14.804540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:14.804555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:14.804570 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:14.804638 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:14.804661 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:14.804797 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:14.804865 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:00:14.805113 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260620 09:00:14.806789 5075 log.cc:824] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:14.808785 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: No bootstrap required, opened a new log
I20260620 09:00:14.812155 5075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260620 09:00:14.812358 5075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:14.812391 5075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e19109f4ad4e3c81896576fa17d178, State: Initialized, Role: FOLLOWER
I20260620 09:00:14.812835 5075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260620 09:00:14.812963 5075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:14.813009 5075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:14.813078 5075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:14.814066 5075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260620 09:00:14.814363 5075 leader_election.cc:302] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e19109f4ad4e3c81896576fa17d178; no voters:
I20260620 09:00:14.814600 5075 leader_election.cc:288] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:14.814706 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:14.814941 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [term 1 LEADER]: Becoming Leader. State: Replica: 27e19109f4ad4e3c81896576fa17d178, State: Running, Role: LEADER
I20260620 09:00:14.815261 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260620 09:00:14.815860 5075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:14.817044 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e19109f4ad4e3c81896576fa17d178. Latest consensus state: current_term: 1 leader_uuid: "27e19109f4ad4e3c81896576fa17d178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } } }
I20260620 09:00:14.817070 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e19109f4ad4e3c81896576fa17d178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e19109f4ad4e3c81896576fa17d178" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } } }
I20260620 09:00:14.817193 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:14.817194 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:14.817889 5089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:14.820757 5089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:14.825935 5089 catalog_manager.cc:1347] Generated new cluster ID: 167446edbd924131b956c3599bd6e93c
I20260620 09:00:14.826083 5089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:14.861430 5089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:14.861794 5089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:14.870115 5089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: Generated new TSK 0
I20260620 09:00:14.870841 5089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:14.878883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:00:14.989459 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:14.989733 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:14.989773 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:14.993870 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:14.993922 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:14.993975 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:14.998752 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:14.999816 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:15.001269 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:15.011010 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.011006 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.011080 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.011358 5099 server_base.cc:1034] running on GCE node
I20260620 09:00:15.011785 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:15.012364 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:15.013514 5099 hybrid_clock.cc:648] HybridClock initialized: now 1781946015013482 us; error 47 us; skew 500 ppm
I20260620 09:00:15.015336 5099 webserver.cc:460] Webserver started at http://127.0.98.1:33329/ using document root <none> and password file <none>
I20260620 09:00:15.015964 5099 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:15.016042 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:15.016263 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:15.018520 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "88df281b30c54b5cae1c8a6ddf944237"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "3007f51051e880b9c7acc398ee583e40"
server_key_iv: "64109e4984dad499427df0cd6a016220"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.019102 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88df281b30c54b5cae1c8a6ddf944237"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "3007f51051e880b9c7acc398ee583e40"
server_key_iv: "64109e4984dad499427df0cd6a016220"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.022718 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:00:15.025214 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.026372 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:15.026504 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "88df281b30c54b5cae1c8a6ddf944237"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "3007f51051e880b9c7acc398ee583e40"
server_key_iv: "64109e4984dad499427df0cd6a016220"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.026615 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:15.030999 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:15.031564 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:15.031740 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:15.032286 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:15.033236 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:15.033318 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.033382 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:15.033434 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.042026 5099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38163
I20260620 09:00:15.042047 5226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38163 every 8 connection(s)
I20260620 09:00:15.042886 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:15.045245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5099
I20260620 09:00:15.045379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:15.045696 392 external_mini_cluster.cc:1428] Setting key 1a2ddf3a7bc2aa93ed86e9b2c472146a
I20260620 09:00:15.048774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 09:00:15.049813 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.044720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58783 (local address 127.0.98.62:39947)
0620 09:00:15.044854 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.044864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.046174 (+ 1310us) server_negotiation.cc:407] Connection header received
0620 09:00:15.047701 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.047704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.047771 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.047881 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.049301 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.049307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.049313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.049330 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.049358 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.049371 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.049475 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.049575 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.049628 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:00:15.050547 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.045205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.1:58783)
0620 09:00:15.045977 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.046016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.047445 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.048177 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.048188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.048301 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.049023 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.049025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.049083 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.049152 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.049457 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.049461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.049465 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.049585 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.049770 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260620 09:00:15.051702 5227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260620 09:00:15.051914 5227 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:15.052330 5227 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260620 09:00:15.053871 5039 ts_manager.cc:194] Registered new tserver with Master: 88df281b30c54b5cae1c8a6ddf944237 (127.0.98.1:38163)
W20260620 09:00:15.159313 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:15.159586 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:15.159631 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:15.163957 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:15.164018 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:15.164122 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:15.169025 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:15.170112 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:15.171516 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:15.181185 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.181185 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.181195 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.181444 5230 server_base.cc:1034] running on GCE node
I20260620 09:00:15.182018 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:15.182654 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:15.183815 5230 hybrid_clock.cc:648] HybridClock initialized: now 1781946015183796 us; error 28 us; skew 500 ppm
I20260620 09:00:15.185763 5230 webserver.cc:460] Webserver started at http://127.0.98.2:46687/ using document root <none> and password file <none>
I20260620 09:00:15.186352 5230 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:15.186412 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:15.186627 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:15.188771 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a7763dc3e054967aa94262609c982cb"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "906fb7e826c3b853e7f1100589d656c0"
server_key_iv: "88fdbf3283203214d2cd03fa9fd81044"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.189334 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a7763dc3e054967aa94262609c982cb"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "906fb7e826c3b853e7f1100589d656c0"
server_key_iv: "88fdbf3283203214d2cd03fa9fd81044"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.192948 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:00:15.195438 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.196751 5230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:15.196885 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "3a7763dc3e054967aa94262609c982cb"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "906fb7e826c3b853e7f1100589d656c0"
server_key_iv: "88fdbf3283203214d2cd03fa9fd81044"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.196997 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:15.201622 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:15.202227 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:15.202412 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:15.202948 5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:15.203843 5230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:15.203926 5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.203990 5230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:15.204037 5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.212812 5230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34933
I20260620 09:00:15.212838 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34933 every 8 connection(s)
I20260620 09:00:15.213681 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:15.215142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5230
I20260620 09:00:15.215273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:15.215612 392 external_mini_cluster.cc:1428] Setting key ba459dc20ce99279cddb3a2fa3fc7cea
I20260620 09:00:15.217622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 09:00:15.219923 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.215437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37011 (local address 127.0.98.62:39947)
0620 09:00:15.215603 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.215608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.216751 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 09:00:15.218173 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.218176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.218219 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.218303 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.219482 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.219485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.219488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.219500 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.219521 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.219529 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.219607 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.219780 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.219815 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:00:15.221009 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.215856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.2:37011)
0620 09:00:15.216598 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.216649 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.217937 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.218480 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.218486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.218575 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.219161 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.219165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.219222 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.219318 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.219617 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.219630 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.219638 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.219829 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.220188 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260620 09:00:15.222180 5358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260620 09:00:15.222416 5358 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:15.222808 5358 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260620 09:00:15.223865 5039 ts_manager.cc:194] Registered new tserver with Master: 3a7763dc3e054967aa94262609c982cb (127.0.98.2:34933)
W20260620 09:00:15.330780 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:15.331061 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:15.331104 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:15.335286 5361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:15.335345 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:15.335403 5361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:15.340238 5361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:15.341374 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:15.342787 5361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:15.352521 5366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.352694 5361 server_base.cc:1034] running on GCE node
W20260620 09:00:15.352730 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.352484 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.353220 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:15.353855 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:15.355043 5361 hybrid_clock.cc:648] HybridClock initialized: now 1781946015355012 us; error 43 us; skew 500 ppm
I20260620 09:00:15.356861 5361 webserver.cc:460] Webserver started at http://127.0.98.3:34589/ using document root <none> and password file <none>
I20260620 09:00:15.357434 5361 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:15.357523 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:15.357764 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:15.359750 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "943fb1d9c9cd47e4b73b5887b953d450"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "6286e53680da41e20e5ddda74ab42ab8"
server_key_iv: "1876a14dd59969a3f4ecf1b18026ecf1"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.360338 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "943fb1d9c9cd47e4b73b5887b953d450"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "6286e53680da41e20e5ddda74ab42ab8"
server_key_iv: "1876a14dd59969a3f4ecf1b18026ecf1"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.363883 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:15.366277 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.367617 5361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:00:15.367758 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "943fb1d9c9cd47e4b73b5887b953d450"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "6286e53680da41e20e5ddda74ab42ab8"
server_key_iv: "1876a14dd59969a3f4ecf1b18026ecf1"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.367873 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:15.372586 5361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:15.373196 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:15.373390 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:15.373919 5361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:15.374799 5361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:15.374871 5361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.374938 5361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:15.374991 5361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.385248 5361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36349
I20260620 09:00:15.385289 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36349 every 8 connection(s)
I20260620 09:00:15.386312 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:15.392428 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.388265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36501 (local address 127.0.98.62:39947)
0620 09:00:15.388408 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.388413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.389551 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 09:00:15.390848 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.390852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.390897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.390994 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.392032 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.392035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.392037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.392049 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.392068 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.392079 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.392158 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.392284 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.392320 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:00:15.393368 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.388765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.3:36501)
0620 09:00:15.389400 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.389441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.390611 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.391184 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.391191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.391281 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.391783 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.391785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.391824 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.391892 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.392160 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.392167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.392171 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.392305 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.392583 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260620 09:00:15.394156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5361
I20260620 09:00:15.394294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:15.394575 392 external_mini_cluster.cc:1428] Setting key 48accf1caaf06bc82477f78d609e0092
I20260620 09:00:15.394594 5489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260620 09:00:15.394820 5489 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:15.395308 5489 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260620 09:00:15.396430 5038 ts_manager.cc:194] Registered new tserver with Master: 943fb1d9c9cd47e4b73b5887b953d450 (127.0.98.3:36349)
I20260620 09:00:15.408643 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:00:15.411893 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.410464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.62:39947)
0620 09:00:15.410601 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.410605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.410745 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:00:15.410853 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.410855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.410888 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.410956 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.411587 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.411590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.411593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.411603 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.411622 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.411630 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.411701 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.411760 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.411793 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:00:15.414868 5038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:15.418102 5038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:15.429553 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.426610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34933 (local address 127.0.0.1:59380)
0620 09:00:15.427026 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.427042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.427122 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.427997 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.428003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.428087 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.428899 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.428902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.428917 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.428948 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.429279 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.429282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.429285 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.429362 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.429397 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":101,"threads_started":1}
I20260620 09:00:15.429615 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.426917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.98.2:34933)
0620 09:00:15.427532 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.427548 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.427571 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:00:15.427626 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.427634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.427839 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.428000 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.429055 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.429061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.429069 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.429080 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.429131 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.429163 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.429298 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.429373 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.429449 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":141,"threads_started":1}
I20260620 09:00:15.429852 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.426236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36349 (local address 127.0.0.1:54856)
0620 09:00:15.426607 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.426671 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.427005 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.427998 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.428005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.428087 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.428787 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.428788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.428804 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.428855 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.429198 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.429207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.429213 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.429322 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.429392 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260620 09:00:15.430153 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.426376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.0.98.3:36349)
0620 09:00:15.427532 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.427544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.427570 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:00:15.427625 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.427631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.427838 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.427998 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.428981 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.428989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.429001 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.429013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.429056 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.429103 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.429217 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.429478 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.429547 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":150,"threads_started":1}
I20260620 09:00:15.431531 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.426997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38163 (local address 127.0.0.1:38240)
0620 09:00:15.428883 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.428896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.428974 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.429852 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.429856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.430101 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.430858 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.430860 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.430871 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.430899 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.431291 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.431294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.431297 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.431376 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.431413 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1807,"thread_start_us":104,"threads_started":1}
I20260620 09:00:15.431779 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.427387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.98.1:38163)
0620 09:00:15.427752 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.427761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.429075 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 09:00:15.429135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.429138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.429332 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.429450 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.431033 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.431042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.431054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.431067 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.431110 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.431147 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.431249 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.431425 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.431503 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":89,"threads_started":1}
I20260620 09:00:15.433037 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc0dddffaac47be98efc4883b420a49 (DEFAULT_TABLE table=test-table [id=bcd393afc0e94d959007062747e456ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:15.433058 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc0dddffaac47be98efc4883b420a49 (DEFAULT_TABLE table=test-table [id=bcd393afc0e94d959007062747e456ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:15.433933 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc0dddffaac47be98efc4883b420a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:15.434044 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc0dddffaac47be98efc4883b420a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:15.434320 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc0dddffaac47be98efc4883b420a49 (DEFAULT_TABLE table=test-table [id=bcd393afc0e94d959007062747e456ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:15.435223 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc0dddffaac47be98efc4883b420a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:15.439651 5506 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Bootstrap starting.
I20260620 09:00:15.441679 5507 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Bootstrap starting.
I20260620 09:00:15.442015 5506 tablet_bootstrap.cc:654] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:15.442800 5506 log.cc:824] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:15.443662 5507 tablet_bootstrap.cc:654] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:15.444303 5507 log.cc:824] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:15.444893 5506 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: No bootstrap required, opened a new log
I20260620 09:00:15.444896 5508 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: Bootstrap starting.
I20260620 09:00:15.445127 5506 ts_tablet_manager.cc:1397] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:00:15.446236 5507 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: No bootstrap required, opened a new log
I20260620 09:00:15.446437 5507 ts_tablet_manager.cc:1397] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:00:15.446743 5508 tablet_bootstrap.cc:654] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:15.447367 5508 log.cc:824] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:15.449169 5508 tablet_bootstrap.cc:492] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: No bootstrap required, opened a new log
I20260620 09:00:15.449371 5508 ts_tablet_manager.cc:1397] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:00:15.449723 5506 raft_consensus.cc:348] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.449994 5506 raft_consensus.cc:374] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:15.450057 5506 raft_consensus.cc:729] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7763dc3e054967aa94262609c982cb, State: Initialized, Role: FOLLOWER
I20260620 09:00:15.450536 5506 consensus_queue.cc:260] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.450920 5507 raft_consensus.cc:348] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.451149 5507 raft_consensus.cc:374] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:15.451203 5507 raft_consensus.cc:729] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943fb1d9c9cd47e4b73b5887b953d450, State: Initialized, Role: FOLLOWER
I20260620 09:00:15.451331 5358 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
I20260620 09:00:15.451575 5506 ts_tablet_manager.cc:1428] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:00:15.451637 5507 consensus_queue.cc:260] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.452483 5489 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
I20260620 09:00:15.452642 5507 ts_tablet_manager.cc:1428] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:00:15.454182 5508 raft_consensus.cc:348] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.454433 5508 raft_consensus.cc:374] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:15.454485 5508 raft_consensus.cc:729] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88df281b30c54b5cae1c8a6ddf944237, State: Initialized, Role: FOLLOWER
I20260620 09:00:15.454886 5508 consensus_queue.cc:260] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.455473 5227 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
I20260620 09:00:15.455798 5508 ts_tablet_manager.cc:1428] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 09:00:15.464641 5359 tablet.cc:2369] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:15.470178 5514 raft_consensus.cc:482] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:15.470395 5514 raft_consensus.cc:504] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.471508 5514 leader_election.cc:288] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 943fb1d9c9cd47e4b73b5887b953d450 (127.0.98.3:36349), 3a7763dc3e054967aa94262609c982cb (127.0.98.2:34933)
I20260620 09:00:15.473409 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.471705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36349 (local address 127.0.98.1:45597)
0620 09:00:15.471834 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.471851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.471932 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.472209 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.472212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.472228 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.472804 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.472806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.472820 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.472855 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.473140 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.473144 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.473148 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.473236 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.473274 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 09:00:15.473436 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.471804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45597 (local address 127.0.98.3:36349)
0620 09:00:15.471965 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.471971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.471988 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:00:15.472042 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.472046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.472091 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.472186 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.472978 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.472981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.472984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.472997 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.473018 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.473027 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.473127 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.473241 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.473275 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 09:00:15.473599 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.471874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46577 (local address 127.0.98.2:34933)
0620 09:00:15.472008 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.472013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.472280 (+ 267us) server_negotiation.cc:407] Connection header received
0620 09:00:15.472394 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.472400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.472439 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.472540 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.473189 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.473195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.473200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.473215 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.473239 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.473250 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.473332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.473420 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.473462 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:00:15.473605 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.471705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34933 (local address 127.0.98.1:46577)
0620 09:00:15.472181 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.472192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.472303 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.472563 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.472566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.472581 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:15.472996 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:15.472997 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:15.473008 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:15.473064 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:15.473337 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:15.473340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:15.473343 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:00:15.473421 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:00:15.473455 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":107,"thread_start_us":145,"threads_started":1}
I20260620 09:00:15.474170 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc0dddffaac47be98efc4883b420a49" candidate_uuid: "88df281b30c54b5cae1c8a6ddf944237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7763dc3e054967aa94262609c982cb" is_pre_election: true
I20260620 09:00:15.474303 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc0dddffaac47be98efc4883b420a49" candidate_uuid: "88df281b30c54b5cae1c8a6ddf944237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "943fb1d9c9cd47e4b73b5887b953d450" is_pre_election: true
I20260620 09:00:15.474472 5312 raft_consensus.cc:2454] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88df281b30c54b5cae1c8a6ddf944237 in term 0.
I20260620 09:00:15.474519 5443 raft_consensus.cc:2454] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88df281b30c54b5cae1c8a6ddf944237 in term 0.
I20260620 09:00:15.474898 5116 leader_election.cc:302] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7763dc3e054967aa94262609c982cb, 88df281b30c54b5cae1c8a6ddf944237; no voters:
I20260620 09:00:15.475198 5514 raft_consensus.cc:2781] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:15.475281 5514 raft_consensus.cc:482] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:15.475311 5514 raft_consensus.cc:3037] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:15.476351 5514 raft_consensus.cc:504] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.476738 5514 leader_election.cc:288] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [CANDIDATE]: Term 1 election: Requested vote from peers 943fb1d9c9cd47e4b73b5887b953d450 (127.0.98.3:36349), 3a7763dc3e054967aa94262609c982cb (127.0.98.2:34933)
I20260620 09:00:15.477015 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc0dddffaac47be98efc4883b420a49" candidate_uuid: "88df281b30c54b5cae1c8a6ddf944237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "943fb1d9c9cd47e4b73b5887b953d450"
I20260620 09:00:15.477062 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc0dddffaac47be98efc4883b420a49" candidate_uuid: "88df281b30c54b5cae1c8a6ddf944237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7763dc3e054967aa94262609c982cb"
I20260620 09:00:15.477131 5443 raft_consensus.cc:3037] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:15.477200 5312 raft_consensus.cc:3037] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:15.478168 5443 raft_consensus.cc:2454] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88df281b30c54b5cae1c8a6ddf944237 in term 1.
I20260620 09:00:15.478257 5312 raft_consensus.cc:2454] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88df281b30c54b5cae1c8a6ddf944237 in term 1.
I20260620 09:00:15.478456 5114 leader_election.cc:302] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88df281b30c54b5cae1c8a6ddf944237, 943fb1d9c9cd47e4b73b5887b953d450; no voters:
I20260620 09:00:15.478669 5514 raft_consensus.cc:2781] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:15.478895 5514 raft_consensus.cc:686] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [term 1 LEADER]: Becoming Leader. State: Replica: 88df281b30c54b5cae1c8a6ddf944237, State: Running, Role: LEADER
I20260620 09:00:15.479197 5514 consensus_queue.cc:237] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } }
I20260620 09:00:15.482594 5039 catalog_manager.cc:5495] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88df281b30c54b5cae1c8a6ddf944237 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "88df281b30c54b5cae1c8a6ddf944237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88df281b30c54b5cae1c8a6ddf944237" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:15.498562 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.497189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.98.1:38163)
0620 09:00:15.497306 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.497309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.497322 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:15.497390 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.497392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.497447 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.497516 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.498125 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.498128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.498130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.498141 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.498158 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.498166 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.498262 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.498345 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.498394 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:00:15.504565 5312 raft_consensus.cc:1261] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 1 FOLLOWER]: Refusing update from remote peer 88df281b30c54b5cae1c8a6ddf944237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:15.504604 5443 raft_consensus.cc:1261] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 1 FOLLOWER]: Refusing update from remote peer 88df281b30c54b5cae1c8a6ddf944237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:15.505329 5514 consensus_queue.cc:1035] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943fb1d9c9cd47e4b73b5887b953d450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:15.505581 5516 consensus_queue.cc:1035] T 0fc0dddffaac47be98efc4883b420a49 P 88df281b30c54b5cae1c8a6ddf944237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7763dc3e054967aa94262609c982cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:15.513289 5520 mvcc.cc:205] Tried to move back new op lower bound from 7298850879498915840 to 7298850879405522944. Current Snapshot: MvccSnapshot[applied={T|T < 7298850879498915840}]
I20260620 09:00:15.515331 5521 mvcc.cc:205] Tried to move back new op lower bound from 7298850879498915840 to 7298850879405522944. Current Snapshot: MvccSnapshot[applied={T|T < 7298850879498915840}]
I20260620 09:00:15.521311 5038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:15.521488 5038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:15.523751 5038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27e19109f4ad4e3c81896576fa17d178: Sending DeleteTablet for 3 replicas of tablet 0fc0dddffaac47be98efc4883b420a49
I20260620 09:00:15.524607 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc0dddffaac47be98efc4883b420a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:15 UTC) from {username='slave'} at 127.0.0.1:59380
I20260620 09:00:15.524758 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc0dddffaac47be98efc4883b420a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:15 UTC) from {username='slave'} at 127.0.0.1:54856
I20260620 09:00:15.525056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5099
I20260620 09:00:15.526721 5530 tablet_replica.cc:330] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: stopping tablet replica
I20260620 09:00:15.527029 5530 raft_consensus.cc:2229] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:15.527320 5530 raft_consensus.cc:2258] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:15.527660 5529 tablet_replica.cc:330] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: stopping tablet replica
I20260620 09:00:15.527925 5529 raft_consensus.cc:2229] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:15.528192 5529 raft_consensus.cc:2258] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:15.528801 5530 ts_tablet_manager.cc:1905] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:15.529645 5529 ts_tablet_manager.cc:1905] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:15.531002 5530 ts_tablet_manager.cc:1918] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:15.531121 5530 log.cc:1196] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/0fc0dddffaac47be98efc4883b420a49
I20260620 09:00:15.531392 5530 ts_tablet_manager.cc:1939] T 0fc0dddffaac47be98efc4883b420a49 P 943fb1d9c9cd47e4b73b5887b953d450: Deleting consensus metadata
I20260620 09:00:15.532269 5529 ts_tablet_manager.cc:1918] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:15.532366 5529 log.cc:1196] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/0fc0dddffaac47be98efc4883b420a49
W20260620 09:00:15.532341 5023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38163 (error 108)
I20260620 09:00:15.532739 5529 ts_tablet_manager.cc:1939] T 0fc0dddffaac47be98efc4883b420a49 P 3a7763dc3e054967aa94262609c982cb: Deleting consensus metadata
I20260620 09:00:15.532879 5023 catalog_manager.cc:4841] TS 943fb1d9c9cd47e4b73b5887b953d450 (127.0.98.3:36349): tablet 0fc0dddffaac47be98efc4883b420a49 (table test-table [id=bcd393afc0e94d959007062747e456ad]) successfully deleted
W20260620 09:00:15.533481 5504 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:15.533204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38163 (local address 127.0.0.1:38256)
0620 09:00:15.533335 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.533348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.533406 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38163: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 09:00:15.533680 5023 catalog_manager.cc:4584] TS 88df281b30c54b5cae1c8a6ddf944237 (127.0.98.1:38163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fc0dddffaac47be98efc4883b420a49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38163: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 09:00:15.533699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5230
I20260620 09:00:15.535179 5025 catalog_manager.cc:4841] TS 3a7763dc3e054967aa94262609c982cb (127.0.98.2:34933): tablet 0fc0dddffaac47be98efc4883b420a49 (table test-table [id=bcd393afc0e94d959007062747e456ad]) successfully deleted
I20260620 09:00:15.539227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5361
I20260620 09:00:15.544700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5008
2026-06-20T09:00:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (938 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 09:00:15.564985 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:00:15.566531 392 webserver.cc:460] Webserver started at https://127.0.0.1:36633/ using document root <none> and password file <none>
I20260620 09:00:15.566718 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:00:15.566999 392 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
2026-06-20T09:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:15Z Disabled control of system clock
I20260620 09:00:15.575742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33625
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33625
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36633/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/testchainca.cert with env {}
W20260620 09:00:15.683604 5537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:15.683912 5537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:15.683959 5537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:15.687930 5537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:15.687994 5537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:15.688017 5537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:15.688035 5537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:15.688050 5537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:00:15.688066 5537 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:00:15.688079 5537 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36633/jwks.json
I20260620 09:00:15.693869 5537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36633/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:15.694933 5537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:15.695914 5537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:15.704336 5545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:15.704350 5543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.704594 5537 server_base.cc:1034] running on GCE node
W20260620 09:00:15.704356 5542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:15.705286 5537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:15.706367 5537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:15.707552 5537 hybrid_clock.cc:648] HybridClock initialized: now 1781946015707538 us; error 28 us; skew 500 ppm
I20260620 09:00:15.709422 5537 webserver.cc:460] Webserver started at http://127.0.98.62:34261/ using document root <none> and password file <none>
I20260620 09:00:15.709990 5537 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:15.710086 5537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:15.710330 5537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:15.712363 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "06d051522dd94973b86f0a724e314adc"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "f5e5fd728bc207840b922db5ae58704a"
server_key_iv: "da3cf821812b5f93e184540a1f21e0c3"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.712946 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "06d051522dd94973b86f0a724e314adc"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "f5e5fd728bc207840b922db5ae58704a"
server_key_iv: "da3cf821812b5f93e184540a1f21e0c3"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.716418 5537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:15.718782 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:15.720085 5537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:15.720199 5537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "06d051522dd94973b86f0a724e314adc"
format_stamp: "Formatted at 2026-06-20 09:00:15 on dist-test-slave-scsf"
server_key: "f5e5fd728bc207840b922db5ae58704a"
server_key_iv: "da3cf821812b5f93e184540a1f21e0c3"
server_key_version: "encryptionkey@0"
I20260620 09:00:15.720289 5537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:15.740084 5537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:15.740967 5537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:15.741181 5537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:15.748183 5537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33625
I20260620 09:00:15.748214 5603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33625 every 8 connection(s)
I20260620 09:00:15.749197 5537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:15.751178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5537
I20260620 09:00:15.751276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:15.751600 392 external_mini_cluster.cc:1428] Setting key dfcfd758a1e82dae21b8079f84725a60
I20260620 09:00:15.752826 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:15.758276 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc: Bootstrap starting.
I20260620 09:00:15.760360 5604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:15.761138 5604 log.cc:824] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:15.761168 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.753109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.0.98.62:33625)
0620 09:00:15.753776 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.753791 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.753824 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:00:15.754511 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.754528 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.754968 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.755319 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:15.755772 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.756882 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.757844 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.758315 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.758505 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:15.759379 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:15.759383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:15.759395 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:15.759774 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:15.759830 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:15.759850 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:15.759959 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:15.760074 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 09:00:15.760327 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":159,"threads_started":1}
I20260620 09:00:15.762962 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc: No bootstrap required, opened a new log
I20260620 09:00:15.764021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:00:15.766780 5604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260620 09:00:15.767046 5604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:15.767149 5604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d051522dd94973b86f0a724e314adc, State: Initialized, Role: FOLLOWER
I20260620 09:00:15.767553 5604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260620 09:00:15.767700 5604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:15.767781 5604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:15.767875 5604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:15.768796 5604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260620 09:00:15.769173 5604 leader_election.cc:302] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d051522dd94973b86f0a724e314adc; no voters:
I20260620 09:00:15.769488 5604 leader_election.cc:288] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:15.769634 5609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:15.769857 5609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [term 1 LEADER]: Becoming Leader. State: Replica: 06d051522dd94973b86f0a724e314adc, State: Running, Role: LEADER
I20260620 09:00:15.770146 5609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260620 09:00:15.770778 5604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:15.771940 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d051522dd94973b86f0a724e314adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } } }
I20260620 09:00:15.771968 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d051522dd94973b86f0a724e314adc. Latest consensus state: current_term: 1 leader_uuid: "06d051522dd94973b86f0a724e314adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d051522dd94973b86f0a724e314adc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } } }
I20260620 09:00:15.772071 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:15.772083 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:15.772497 5618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:15.775301 5618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:15.780519 5618 catalog_manager.cc:1347] Generated new cluster ID: 94de67cf3a93446faeafe205b5156c52
I20260620 09:00:15.780587 5618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:15.802642 5618 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:15.803996 5618 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:15.810480 5618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d051522dd94973b86f0a724e314adc: Generated new TSK 0
I20260620 09:00:15.811259 5618 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:00:15.841943 5634 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 09:00:15.842073 5634 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:00:15.842610 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:15.825158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.98.62:33625)
0620 09:00:15.825289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:15.825295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:15.825513 (+ 218us) server_negotiation.cc:407] Connection header received
0620 09:00:15.825729 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:15.825733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:15.825789 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:15.825901 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:00:15.826132 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.826966 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.827754 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.828092 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.828191 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:15.828691 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:00:15.842254 (+ 13563us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:00:15.842482 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 09:00:15.842722 5607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 09:00:15.844086 5633 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:15.825050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33625 (local address 127.0.0.1:45416)
0620 09:00:15.825438 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:15.825450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:15.825570 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:15.825915 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:15.825917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:15.825923 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:00:15.826030 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:15.826034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.827065 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.827068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:15.827653 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:15.827659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:15.828266 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:15.828286 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:15.828378 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:15.828432 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:00:15.842490 (+ 14058us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:00:15.842749 (+ 259us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36633/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 09:00:15.845026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5537
2026-06-20T09:00:15Z chronyd exiting
I20260620 09:00:15.882676 392 test_util.cc:175] -----------------------------------------------
I20260620 09:00:15.882766 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781945987698834-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 09:00:15.883754 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:00:15.884620 392 webserver.cc:460] Webserver started at https://127.0.0.1:46511/ using document root <none> and password file <none>
I20260620 09:00:15.884704 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:00:15.884907 392 webserver.cc:460] Webserver started at http://127.0.0.1:45583/ using document root <none> and password file <none>
2026-06-20T09:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:15Z Disabled control of system clock
I20260620 09:00:15.893504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43905
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46511/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/testchainca.cert with env {}
W20260620 09:00:16.003105 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:16.003379 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:16.003418 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:16.007352 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:16.007418 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:16.007434 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:16.007450 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:16.007467 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:00:16.007483 5640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:00:16.007495 5640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46511/jwks.json
I20260620 09:00:16.013013 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46511/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:16.014050 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:16.015036 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:16.024219 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:16.024247 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:16.024225 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:16.024489 5640 server_base.cc:1034] running on GCE node
I20260620 09:00:16.025136 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:16.026276 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:16.027468 5640 hybrid_clock.cc:648] HybridClock initialized: now 1781946016027436 us; error 45 us; skew 500 ppm
I20260620 09:00:16.029361 5640 webserver.cc:460] Webserver started at http://127.0.98.62:36723/ using document root <none> and password file <none>
I20260620 09:00:16.030007 5640 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:16.030098 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:16.030321 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:16.032322 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "8aadb0c6319a4d2dae3a6909c934c25a"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "1ae03138662516ae25d83b67146b4cab"
server_key_iv: "e2a12283c043f7531187df93bde31c60"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.032953 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "8aadb0c6319a4d2dae3a6909c934c25a"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "1ae03138662516ae25d83b67146b4cab"
server_key_iv: "e2a12283c043f7531187df93bde31c60"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.036403 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:16.038775 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:16.039969 5640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:16.040107 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "8aadb0c6319a4d2dae3a6909c934c25a"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "1ae03138662516ae25d83b67146b4cab"
server_key_iv: "e2a12283c043f7531187df93bde31c60"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.040217 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:16.060925 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:16.061640 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:16.061839 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:16.068359 5640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43905
I20260620 09:00:16.068369 5706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43905 every 8 connection(s)
I20260620 09:00:16.069344 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:16.072268 5707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:16.077546 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: Bootstrap starting.
I20260620 09:00:16.079253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5640
I20260620 09:00:16.079372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:16.079664 392 external_mini_cluster.cc:1428] Setting key 30ca1b124c0f3c840ff2114d3e416681
I20260620 09:00:16.080160 5707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:16.081084 5707 log.cc:824] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:16.083195 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: No bootstrap required, opened a new log
I20260620 09:00:16.087355 5707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } }
I20260620 09:00:16.087604 5707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:16.087667 5707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aadb0c6319a4d2dae3a6909c934c25a, State: Initialized, Role: FOLLOWER
I20260620 09:00:16.088188 5707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } }
I20260620 09:00:16.088325 5707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:16.088410 5707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:16.088562 5707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:16.089068 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.080744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.0.98.62:43905)
0620 09:00:16.081401 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.081417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.081477 (+ 60us) server_negotiation.cc:407] Connection header received
0620 09:00:16.082284 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.082303 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.082650 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.082933 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:16.083302 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.084262 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.085498 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.085988 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.086111 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.087029 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:16.087035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:16.087048 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:16.087432 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:16.087487 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:16.087506 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:16.087609 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:16.087785 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:00:16.088101 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":118,"threads_started":1}
I20260620 09:00:16.089632 5707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } }
I20260620 09:00:16.089978 5707 leader_election.cc:302] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aadb0c6319a4d2dae3a6909c934c25a; no voters:
I20260620 09:00:16.090323 5707 leader_election.cc:288] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:16.090425 5712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:16.090667 5712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [term 1 LEADER]: Becoming Leader. State: Replica: 8aadb0c6319a4d2dae3a6909c934c25a, State: Running, Role: LEADER
I20260620 09:00:16.090968 5712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } }
I20260620 09:00:16.091711 5707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:16.092434 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:00:16.092881 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } } }
I20260620 09:00:16.092998 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:16.094558 5714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aadb0c6319a4d2dae3a6909c934c25a. Latest consensus state: current_term: 1 leader_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aadb0c6319a4d2dae3a6909c934c25a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43905 } } }
I20260620 09:00:16.094661 5714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a [sys.catalog]: This master's current role is: LEADER
W20260620 09:00:16.096411 5728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:16.096532 5728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:16.096650 5721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:16.097325 5721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:16.102583 5721 catalog_manager.cc:1347] Generated new cluster ID: e5a3a2373f4246d183dd80590c4af8a9
I20260620 09:00:16.102653 5721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:16.123903 5721 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:16.125224 5721 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:16.133755 5721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aadb0c6319a4d2dae3a6909c934c25a: Generated new TSK 0
I20260620 09:00:16.134611 5721 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:00:16.158710 5737 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 09:00:16.158803 5737 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:00:16.159574 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.143540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.0.98.62:43905)
0620 09:00:16.143707 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.143712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.143975 (+ 263us) server_negotiation.cc:407] Connection header received
0620 09:00:16.144109 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.144112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.144168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.144279 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:00:16.144490 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.145316 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.146204 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.146543 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.146652 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.146898 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:00:16.159134 (+ 12236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:00:16.159415 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 09:00:16.159598 5736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:16.143462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43905 (local address 127.0.0.1:36648)
0620 09:00:16.143886 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:16.143899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:16.143991 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:16.144280 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:16.144282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:16.144288 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:00:16.144371 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.144375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.145442 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.145445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.146096 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.146103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.146638 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.146653 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.146718 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.146739 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:00:16.159324 (+ 12585us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:00:16.159463 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":249,"threads_started":1}
W20260620 09:00:16.159652 5710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 09:00:16.160660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5640
2026-06-20T09:00:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 09:00:16.200169 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:00:16.201090 392 webserver.cc:460] Webserver started at https://127.0.0.1:41071/ using document root <none> and password file <none>
I20260620 09:00:16.201184 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:00:16.201395 392 webserver.cc:460] Webserver started at http://127.0.0.1:34421/ using document root <none> and password file <none>
2026-06-20T09:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:16Z Disabled control of system clock
I20260620 09:00:16.210147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42793
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:00:16.319702 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:16.319973 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:16.320012 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:16.324021 5743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:16.324093 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:16.324110 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:16.324127 5743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:16.324143 5743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:00:16.324159 5743 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:00:16.324173 5743 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41071/jwks.json
I20260620 09:00:16.329097 5743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:16.330179 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:16.331512 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:16.340014 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:16.340091 5743 server_base.cc:1034] running on GCE node
W20260620 09:00:16.340010 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:16.340099 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:16.340880 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:16.342059 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:16.343253 5743 hybrid_clock.cc:648] HybridClock initialized: now 1781946016343240 us; error 27 us; skew 500 ppm
I20260620 09:00:16.345206 5743 webserver.cc:460] Webserver started at http://127.0.98.62:36571/ using document root <none> and password file <none>
I20260620 09:00:16.345808 5743 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:16.345904 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:16.346145 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:16.348201 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "28396788174049868b478344696e0e49"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "a8687ec380def5d181ce25609fcbc337"
server_key_iv: "4da83fb9755fc50294e0fa8608ededc8"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.348903 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "28396788174049868b478344696e0e49"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "a8687ec380def5d181ce25609fcbc337"
server_key_iv: "4da83fb9755fc50294e0fa8608ededc8"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.353312 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:16.355736 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:16.357023 5743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:16.357148 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "28396788174049868b478344696e0e49"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "a8687ec380def5d181ce25609fcbc337"
server_key_iv: "4da83fb9755fc50294e0fa8608ededc8"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.357250 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:16.374825 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:16.375530 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:16.375736 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:16.382743 5743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42793
I20260620 09:00:16.382750 5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42793 every 8 connection(s)
I20260620 09:00:16.383733 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:16.386039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5743
I20260620 09:00:16.386185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:16.386487 392 external_mini_cluster.cc:1428] Setting key 824254e9aaf4dffbabe40f4ab5e1e91d
I20260620 09:00:16.387643 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:16.394662 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49: Bootstrap starting.
I20260620 09:00:16.397028 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.387916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.98.62:42793)
0620 09:00:16.388617 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.388639 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.388682 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:00:16.389522 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.389544 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.390055 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.390447 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:16.390900 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.392378 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.393642 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.394124 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.394247 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.395206 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:16.395211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:16.395223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:16.395606 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:16.395661 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:16.395680 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:16.395756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:16.395930 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 09:00:16.396232 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
I20260620 09:00:16.397243 5810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:16.398006 5810 log.cc:824] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:16.400048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:00:16.400264 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49: No bootstrap required, opened a new log
I20260620 09:00:16.404217 5810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260620 09:00:16.404518 5810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:16.404611 5810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28396788174049868b478344696e0e49, State: Initialized, Role: FOLLOWER
I20260620 09:00:16.405066 5810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260620 09:00:16.405192 5810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:16.405259 5810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:16.405367 5810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:16.406318 5810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260620 09:00:16.406668 5810 leader_election.cc:302] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28396788174049868b478344696e0e49; no voters:
I20260620 09:00:16.407019 5810 leader_election.cc:288] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:16.407182 5815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:16.407428 5815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [term 1 LEADER]: Becoming Leader. State: Replica: 28396788174049868b478344696e0e49, State: Running, Role: LEADER
I20260620 09:00:16.407724 5815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } }
I20260620 09:00:16.408277 5810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:16.409350 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28396788174049868b478344696e0e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } }
I20260620 09:00:16.409480 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:16.409792 5817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28396788174049868b478344696e0e49. Latest consensus state: current_term: 1 leader_uuid: "28396788174049868b478344696e0e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28396788174049868b478344696e0e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } }
I20260620 09:00:16.409881 5817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:16.409979 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:16.413197 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:16.419610 5824 catalog_manager.cc:1347] Generated new cluster ID: 310ef3e4d2d74035971b5a7776b4093f
I20260620 09:00:16.419694 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:16.429800 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:16.430701 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:16.444586 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28396788174049868b478344696e0e49: Generated new TSK 0
I20260620 09:00:16.445350 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:16.458707 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.453517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.0.98.62:42793)
0620 09:00:16.453656 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.453661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.453805 (+ 144us) server_negotiation.cc:407] Connection header received
0620 09:00:16.453936 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.453939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.453995 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.454116 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:00:16.454316 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.455168 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.456004 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.456369 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.456497 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.456802 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:00:16.458316 (+ 1514us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:00:16.458551 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 09:00:16.458782 5813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 09:00:16.460382 5839 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:16.453401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.0.1:42990)
0620 09:00:16.453720 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:16.453732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:16.453828 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:16.454114 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:16.454116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:16.454122 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:00:16.454216 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.454220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.455273 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.455275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.455897 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.455904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.456525 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.456541 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.456613 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.456635 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:00:16.458483 (+ 1848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:00:16.460211 (+ 1728us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42793: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1461,"thread_start_us":97,"threads_started":1}
I20260620 09:00:16.461035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5743
E20260620 09:00:16.469470 5840 webserver.cc:533] Webserver: sslize failed:
E20260620 09:00:16.469547 5840 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T09:00:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T09:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:16Z Disabled control of system clock
I20260620 09:00:16.508287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42737
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42737
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/testchainca.cert with env {}
W20260620 09:00:16.617532 5844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:16.617801 5844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:16.617838 5844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:16.621668 5844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:16.621721 5844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:16.621737 5844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:16.621754 5844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:16.621769 5844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:00:16.621785 5844 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:00:16.621798 5844 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 09:00:16.627039 5844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:16.628046 5844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:16.629041 5844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:00:16.637290 5844 server_base.cc:1034] running on GCE node
W20260620 09:00:16.637245 5850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:16.637245 5852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:16.637264 5849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:16.638156 5844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:16.639209 5844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:16.640395 5844 hybrid_clock.cc:648] HybridClock initialized: now 1781946016640376 us; error 32 us; skew 500 ppm
I20260620 09:00:16.645489 5844 webserver.cc:460] Webserver started at http://127.0.98.62:32937/ using document root <none> and password file <none>
I20260620 09:00:16.646059 5844 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:16.646149 5844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:16.646369 5844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:16.648439 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "501f11fe2ad64abe9bfda173587bc362"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "e1e062cc1913c46f25f0491b3f444bcf"
server_key_iv: "5d6d815c3e12020188e7985b049b6fa5"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.649065 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "501f11fe2ad64abe9bfda173587bc362"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "e1e062cc1913c46f25f0491b3f444bcf"
server_key_iv: "5d6d815c3e12020188e7985b049b6fa5"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.652724 5844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:16.655153 5858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:16.656352 5844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:16.656529 5844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "501f11fe2ad64abe9bfda173587bc362"
format_stamp: "Formatted at 2026-06-20 09:00:16 on dist-test-slave-scsf"
server_key: "e1e062cc1913c46f25f0491b3f444bcf"
server_key_iv: "5d6d815c3e12020188e7985b049b6fa5"
server_key_version: "encryptionkey@0"
I20260620 09:00:16.656646 5844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:16.685140 5844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:16.685878 5844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:16.686081 5844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:16.692690 5844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42737
I20260620 09:00:16.692708 5910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42737 every 8 connection(s)
I20260620 09:00:16.693876 5844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:16.694072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5844
I20260620 09:00:16.694213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:16.694543 392 external_mini_cluster.cc:1428] Setting key cbca48e63339ee450fda6331156e61e5
I20260620 09:00:16.697830 5911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:16.704924 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362: Bootstrap starting.
I20260620 09:00:16.706905 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.696256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.62:42737)
0620 09:00:16.697186 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.697207 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.697255 (+ 48us) server_negotiation.cc:407] Connection header received
0620 09:00:16.697997 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.698018 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.698571 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.698987 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:16.699967 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.701674 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.703200 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.703801 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.703982 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.704884 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:16.704889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:16.704901 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:16.705340 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:16.705419 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:16.705448 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:16.705561 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:16.705741 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 09:00:16.706153 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":191,"threads_started":1}
I20260620 09:00:16.707468 5911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:16.708184 5911 log.cc:824] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:16.709832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:00:16.710270 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362: No bootstrap required, opened a new log
I20260620 09:00:16.713937 5911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } }
I20260620 09:00:16.714184 5911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:16.714273 5911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501f11fe2ad64abe9bfda173587bc362, State: Initialized, Role: FOLLOWER
I20260620 09:00:16.714680 5911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } }
I20260620 09:00:16.714800 5911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:16.714844 5911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:16.714912 5911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:16.715736 5911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } }
I20260620 09:00:16.716028 5911 leader_election.cc:302] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501f11fe2ad64abe9bfda173587bc362; no voters:
I20260620 09:00:16.716289 5911 leader_election.cc:288] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:16.716423 5916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:16.716679 5916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [term 1 LEADER]: Becoming Leader. State: Replica: 501f11fe2ad64abe9bfda173587bc362, State: Running, Role: LEADER
I20260620 09:00:16.716976 5916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } }
I20260620 09:00:16.717538 5911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:16.718603 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501f11fe2ad64abe9bfda173587bc362. Latest consensus state: current_term: 1 leader_uuid: "501f11fe2ad64abe9bfda173587bc362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } } }
I20260620 09:00:16.718747 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:16.719020 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501f11fe2ad64abe9bfda173587bc362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f11fe2ad64abe9bfda173587bc362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42737 } } }
I20260620 09:00:16.719120 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:16.719228 5925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:16.722393 5925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:16.727667 5925 catalog_manager.cc:1347] Generated new cluster ID: 7cce971f6c94444ba0522c181facf0d9
I20260620 09:00:16.727751 5925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:16.736915 5925 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:16.738142 5925 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:16.749675 5925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501f11fe2ad64abe9bfda173587bc362: Generated new TSK 0
I20260620 09:00:16.750622 5925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:16.817209 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:16.768995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.98.62:42737)
0620 09:00:16.769140 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:16.769145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:16.769340 (+ 195us) server_negotiation.cc:407] Connection header received
0620 09:00:16.769504 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:16.769507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:16.769552 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:16.769653 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:00:16.769890 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.770391 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.771637 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.771836 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.771905 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.772248 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:00:16.816820 (+ 44572us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:00:16.817081 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 09:00:16.817305 5914 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 09:00:16.817759 5940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:16.768926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42737 (local address 127.0.0.1:50202)
0620 09:00:16.769241 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:16.769257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:16.769376 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:16.769666 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:16.769669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:16.769677 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:00:16.769800 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.769806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.770495 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.770499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.771506 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:16.771517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:16.771933 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:16.771954 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:16.772041 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:16.772069 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:00:16.817063 (+ 44994us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:00:16.817296 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":127,"threads_started":1}
I20260620 09:00:16.818389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5844
2026-06-20T09:00:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (337 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:17 dist-test-slave-scsf krb5kdc[5952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:17 dist-test-slave-scsf krb5kdc[5952](info): set up 2 sockets
Jun 20 09:00:17 dist-test-slave-scsf krb5kdc[5952](info): commencing operation
krb5kdc: starting...
W20260620 09:00:18.084916 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:18.203549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38931
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:18.311571 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:18.311883 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:18.311956 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:18.315999 5965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:18.316109 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:18.316150 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:18.316187 5965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:18.316217 5965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:18.321583 5965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:18.322681 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:18.323721 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:18.333181 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:18.333331 5965 server_base.cc:1034] running on GCE node
W20260620 09:00:18.333505 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:18.333189 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:18.334187 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:18.335332 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:18.336524 5965 hybrid_clock.cc:648] HybridClock initialized: now 1781946018336502 us; error 36 us; skew 500 ppm
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:18.339246 5965 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:18.340137 5965 webserver.cc:460] Webserver started at http://127.0.98.62:39025/ using document root <none> and password file <none>
I20260620 09:00:18.340775 5965 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:18.340865 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:18.341123 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:18.343233 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "83bdf532a0c34c4380a20f7a4cb34ab8"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "de80de65bd61be1a3535eec831db9929"
server_key_iv: "64f61ea4664d368f6f72080416a50ded"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.343819 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "83bdf532a0c34c4380a20f7a4cb34ab8"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "de80de65bd61be1a3535eec831db9929"
server_key_iv: "64f61ea4664d368f6f72080416a50ded"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.347554 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:00:18.350003 5980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.351194 5965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:18.351351 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "83bdf532a0c34c4380a20f7a4cb34ab8"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "de80de65bd61be1a3535eec831db9929"
server_key_iv: "64f61ea4664d368f6f72080416a50ded"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.351464 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:18.373145 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:18.376196 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:18.376422 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:18.384029 5965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38931
I20260620 09:00:18.384418 6032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38931 every 8 connection(s)
I20260620 09:00:18.385212 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:18.387964 6033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:18.389576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 5965
I20260620 09:00:18.389720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:18.390013 392 external_mini_cluster.cc:1428] Setting key f4aaf44f974b94301f1fc4e21bf1b303
I20260620 09:00:18.394105 6033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8: Bootstrap starting.
I20260620 09:00:18.396519 6033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:18.397384 6033 log.cc:824] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:18.399713 6033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8: No bootstrap required, opened a new log
I20260620 09:00:18.403848 6033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260620 09:00:18.404071 6033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:18.404114 6033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bdf532a0c34c4380a20f7a4cb34ab8, State: Initialized, Role: FOLLOWER
I20260620 09:00:18.404554 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:18.391427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.98.62:38931)
0620 09:00:18.391868 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:18.391884 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:18.391919 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:00:18.392632 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:18.392650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:18.392991 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:18.393331 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:18.394134 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.395171 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.396024 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.396604 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.396806 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:18.399545 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:18.399551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:18.399564 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:18.399590 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:18.401407 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.401992 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.401997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.402002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.402128 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.402437 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.402439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.402440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.402830 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:18.402975 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:18.403299 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 09:00:18.403640 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260620 09:00:18.404736 6033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260620 09:00:18.404904 6033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:18.404968 6033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:18.405071 6033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:18.406167 6033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260620 09:00:18.406541 6033 leader_election.cc:302] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83bdf532a0c34c4380a20f7a4cb34ab8; no voters:
I20260620 09:00:18.406908 6033 leader_election.cc:288] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:18.407024 6038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:18.407229 6038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 83bdf532a0c34c4380a20f7a4cb34ab8, State: Running, Role: LEADER
I20260620 09:00:18.407531 6038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } }
I20260620 09:00:18.408129 6033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:18.409176 6039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bdf532a0c34c4380a20f7a4cb34ab8. Latest consensus state: current_term: 1 leader_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } } }
I20260620 09:00:18.409324 6039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:18.409212 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdf532a0c34c4380a20f7a4cb34ab8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38931 } } }
I20260620 09:00:18.409545 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:18.409709 6047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:18.412642 6047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:18.417726 6047 catalog_manager.cc:1347] Generated new cluster ID: 4b757812be914201a3b6ec2de295d72e
I20260620 09:00:18.417798 6047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:18.425915 6047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:18.426885 6047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:18.435292 6047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83bdf532a0c34c4380a20f7a4cb34ab8: Generated new TSK 0
I20260620 09:00:18.435971 6047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:18.465534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:18.577401 6059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:18.577673 6059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:18.577718 6059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:18.581849 6059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:18.581916 6059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:18.581976 6059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:18.587357 6059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:18.588421 6059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:18.589532 6059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:18.599047 6064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:18.599050 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:18.599102 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:18.599340 6059 server_base.cc:1034] running on GCE node
I20260620 09:00:18.599740 6059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:18.600346 6059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:18.601529 6059 hybrid_clock.cc:648] HybridClock initialized: now 1781946018601510 us; error 29 us; skew 500 ppm
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:18.604315 6059 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:18.605355 6059 webserver.cc:460] Webserver started at http://127.0.98.1:43787/ using document root <none> and password file <none>
I20260620 09:00:18.605964 6059 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:18.606057 6059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:18.606287 6059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:18.608384 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "69e40b00a1864c6f9e0b26c1555955a9"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "3485fb94998fc24e1607e7b5e51e360a"
server_key_iv: "c1f0c5f3ba9ea0d6ec128eceb0fe393d"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.609042 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "69e40b00a1864c6f9e0b26c1555955a9"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "3485fb94998fc24e1607e7b5e51e360a"
server_key_iv: "c1f0c5f3ba9ea0d6ec128eceb0fe393d"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.612851 6059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:18.615182 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.616315 6059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:18.616492 6059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "69e40b00a1864c6f9e0b26c1555955a9"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "3485fb94998fc24e1607e7b5e51e360a"
server_key_iv: "c1f0c5f3ba9ea0d6ec128eceb0fe393d"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.616611 6059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:18.628407 6059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:18.631800 6059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:18.632015 6059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:18.632804 6059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:18.633879 6059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:18.633950 6059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.634012 6059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:18.634042 6059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.642645 6059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37857
I20260620 09:00:18.642666 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37857 every 8 connection(s)
I20260620 09:00:18.643537 6059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:18.651948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6059
I20260620 09:00:18.652099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:18.652408 392 external_mini_cluster.cc:1428] Setting key 1eafd1beb3a5e8643c2dcd9fcf341c20
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:18.657267 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:18.645315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56729 (local address 127.0.98.62:38931)
0620 09:00:18.645478 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:18.645483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:18.646302 (+ 819us) server_negotiation.cc:407] Connection header received
0620 09:00:18.647587 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:18.647590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:18.647634 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:18.647708 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:18.648958 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.649480 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.650548 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.650766 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.650839 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:18.653306 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:18.653310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:18.653313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:18.653337 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:18.654966 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.655676 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.655681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.655684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.655785 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.656143 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.656148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.656150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.656337 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:18.656447 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:18.657006 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 09:00:18.657109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:00:18.658525 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:18.645715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38931 (local address 127.0.98.1:56729)
0620 09:00:18.646151 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:18.646193 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:18.647400 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:18.647884 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:18.647891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:18.648239 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:18.648810 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:18.648830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.649598 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.649601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:18.650447 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:18.650456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.650891 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.650894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:18.651027 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:18.651552 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:18.651567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:18.653138 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:18.655074 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:18.655083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:18.655109 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:18.655491 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:18.655880 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:18.655882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:18.655883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:18.656015 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:18.656460 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:18.656470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:18.656716 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 09:00:18.656952 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 09:00:18.657352 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260620 09:00:18.659776 6188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38931
I20260620 09:00:18.660045 6188 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:18.660672 6188 heartbeater.cc:511] Master 127.0.98.62:38931 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:18.662261 5997 ts_manager.cc:194] Registered new tserver with Master: 69e40b00a1864c6f9e0b26c1555955a9 (127.0.98.1:37857)
I20260620 09:00:18.663503 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:18.679224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:18.790107 6193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:18.790403 6193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:18.790477 6193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:18.794523 6193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:18.794623 6193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:18.794721 6193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:18.799922 6193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:18.801009 6193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:18.802083 6193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:18.811704 6201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:18.811699 6199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:18.811748 6193 server_base.cc:1034] running on GCE node
W20260620 09:00:18.811699 6198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:18.812345 6193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:18.813035 6193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:18.814188 6193 hybrid_clock.cc:648] HybridClock initialized: now 1781946018814155 us; error 46 us; skew 500 ppm
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:18.816936 6193 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:18.817723 6193 webserver.cc:460] Webserver started at http://127.0.98.2:38081/ using document root <none> and password file <none>
I20260620 09:00:18.818238 6193 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:18.818295 6193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:18.818452 6193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:18.820673 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "61c08fa99782435baaf3e4d7d255c6cf"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "6afd80db3479a1beea02a59b3a8ff13f"
server_key_iv: "4adee504266ea5f8e5b9f48be9b3b382"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.821249 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61c08fa99782435baaf3e4d7d255c6cf"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "6afd80db3479a1beea02a59b3a8ff13f"
server_key_iv: "4adee504266ea5f8e5b9f48be9b3b382"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.824833 6193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:18.827245 6208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.828495 6193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:18.828625 6193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "61c08fa99782435baaf3e4d7d255c6cf"
format_stamp: "Formatted at 2026-06-20 09:00:18 on dist-test-slave-scsf"
server_key: "6afd80db3479a1beea02a59b3a8ff13f"
server_key_iv: "4adee504266ea5f8e5b9f48be9b3b382"
server_key_version: "encryptionkey@0"
I20260620 09:00:18.828729 6193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:18.847844 6193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:18.850881 6193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:18.851086 6193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:18.851688 6193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:18.852581 6193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:18.852648 6193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.852715 6193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:18.852766 6193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:18.861479 6193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46447
I20260620 09:00:18.861501 6321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46447 every 8 connection(s)
I20260620 09:00:18.862386 6193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:18.864899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6193
I20260620 09:00:18.865023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:18.865336 392 external_mini_cluster.cc:1428] Setting key 40d7aaf11e538b94c0288fb110a5db15
Jun 20 09:00:18 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:18.876636 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:18.864183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56307 (local address 127.0.98.62:38931)
0620 09:00:18.864329 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:18.864332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:18.865331 (+ 999us) server_negotiation.cc:407] Connection header received
0620 09:00:18.867084 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:18.867087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:18.867136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:18.867228 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:18.868478 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.868991 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.870194 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.870415 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.870527 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:18.873250 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:18.873253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:18.873255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:18.873277 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:18.874737 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.875192 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.875195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.875196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.875263 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:18.875612 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:18.875615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:18.875616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:18.875785 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:18.875860 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:18.876366 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 09:00:18.876514 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:18.877640 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:18.864608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38931 (local address 127.0.98.2:56307)
0620 09:00:18.865187 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:18.865230 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:18.866864 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:18.867434 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:18.867441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:18.867808 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:18.868317 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:18.868330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.869115 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.869118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:18.870064 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:18.870073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:18.870565 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:18.870569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:18.870749 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:18.871422 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:18.871437 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:18.873054 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:18.874841 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:18.874845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:18.874857 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:18.875111 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:18.875407 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:18.875409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:18.875410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:18.875531 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:18.875869 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:18.875874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:18.876117 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 09:00:18.876328 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 09:00:18.876634 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":155,"threads_started":1}
I20260620 09:00:18.878868 6322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38931
I20260620 09:00:18.879117 6322 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:18.879601 6322 heartbeater.cc:511] Master 127.0.98.62:38931 requested a full tablet report, sending...
I20260620 09:00:18.880743 5997 ts_manager.cc:194] Registered new tserver with Master: 61c08fa99782435baaf3e4d7d255c6cf (127.0.98.2:46447)
I20260620 09:00:18.881343 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:18.891812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:19.001739 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:19.002043 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:19.002116 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:19.006299 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:19.006383 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:19.006484 6327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:19.011958 6327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:19.013031 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:19.014045 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:19.023478 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:19.023466 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:19.023494 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:19.023849 6327 server_base.cc:1034] running on GCE node
I20260620 09:00:19.024248 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:19.024946 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:19.026131 6327 hybrid_clock.cc:648] HybridClock initialized: now 1781946019026115 us; error 28 us; skew 500 ppm
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.028833 6327 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:19.029673 6327 webserver.cc:460] Webserver started at http://127.0.98.3:37327/ using document root <none> and password file <none>
I20260620 09:00:19.030217 6327 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:19.030304 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:19.030522 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:19.032683 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "373f786ea450452abc9944c62575dca1"
format_stamp: "Formatted at 2026-06-20 09:00:19 on dist-test-slave-scsf"
server_key: "019a59292e5366e3a0c417f228107cdb"
server_key_iv: "2e78e5754829aaddd0d2904cf1fc680b"
server_key_version: "encryptionkey@0"
I20260620 09:00:19.033244 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "373f786ea450452abc9944c62575dca1"
format_stamp: "Formatted at 2026-06-20 09:00:19 on dist-test-slave-scsf"
server_key: "019a59292e5366e3a0c417f228107cdb"
server_key_iv: "2e78e5754829aaddd0d2904cf1fc680b"
server_key_version: "encryptionkey@0"
I20260620 09:00:19.036851 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:19.039156 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:19.040328 6327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:19.040495 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "373f786ea450452abc9944c62575dca1"
format_stamp: "Formatted at 2026-06-20 09:00:19 on dist-test-slave-scsf"
server_key: "019a59292e5366e3a0c417f228107cdb"
server_key_iv: "2e78e5754829aaddd0d2904cf1fc680b"
server_key_version: "encryptionkey@0"
I20260620 09:00:19.040629 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:19.059608 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:19.062844 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:19.063055 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:19.063642 6327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:19.064532 6327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:19.064600 6327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:19.064675 6327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:19.064718 6327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:19.073614 6327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39879
I20260620 09:00:19.073637 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39879 every 8 connection(s)
I20260620 09:00:19.074491 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:19.077903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6327
I20260620 09:00:19.078027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:19.078394 392 external_mini_cluster.cc:1428] Setting key 2bb0730304794cc98aee3dd8023a56f1
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:19.087989 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:19.076309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50647 (local address 127.0.98.62:38931)
0620 09:00:19.076494 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.076499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.077309 (+ 810us) server_negotiation.cc:407] Connection header received
0620 09:00:19.079086 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.079089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.079143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.079238 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.080772 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.081302 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.082371 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.082579 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.082707 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:19.085044 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:19.085047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:19.085049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:19.085069 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:19.086280 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.086750 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.086753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.086754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.086822 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.087121 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.087123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.087124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.087282 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:19.087351 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:19.087776 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 09:00:19.087870 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:00:19.088889 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:19.076735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38931 (local address 127.0.98.3:50647)
0620 09:00:19.077166 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:19.077207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:19.078821 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:19.079438 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:19.079448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:19.079972 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:19.080605 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:19.080625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.081405 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.081407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:19.082269 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:19.082278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.082725 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.082728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:19.082870 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:19.083477 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:19.083494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:19.084905 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:19.086417 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:19.086421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:19.086431 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:19.086662 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:19.086891 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:19.086893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:19.086895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:19.087011 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:19.087361 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:19.087366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:19.087577 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 09:00:19.087718 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 09:00:19.087963 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260620 09:00:19.089975 6456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38931
I20260620 09:00:19.090206 6456 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:19.090662 6456 heartbeater.cc:511] Master 127.0.98.62:38931 requested a full tablet report, sending...
I20260620 09:00:19.091714 5997 ts_manager.cc:194] Registered new tserver with Master: 373f786ea450452abc9944c62575dca1 (127.0.98.3:39879)
I20260620 09:00:19.092325 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50647
I20260620 09:00:19.092360 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/test-admin.keytab.
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.109206 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 09:00:19.112003 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:19.110594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.0.98.1:37857)
0620 09:00:19.110954 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.110963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.110988 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:00:19.111037 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.111041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.111229 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.111350 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.111810 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45866: BlockingRecv error: recv got EOF from 127.0.0.1:45866 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":153,"threads_started":1}
W20260620 09:00:19.112099 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.113209 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.114162 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:00:19.123428 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:19.115056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.1:37857)
0620 09:00:19.115187 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.115190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.115260 (+ 70us) server_negotiation.cc:407] Connection header received
0620 09:00:19.115343 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.115345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.115381 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.115447 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.115922 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.116780 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.117583 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.117943 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.118020 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:19.120354 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:19.120363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:19.120376 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:19.120396 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:19.121917 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.122299 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.122303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.122305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.122376 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.122595 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.122598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.122599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.122827 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:19.122984 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:19.123169 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:00:19.123290 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 09:00:19.127204 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:19.126204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.0.98.1:37857)
0620 09:00:19.126325 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.126329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.126436 (+ 107us) server_negotiation.cc:407] Connection header received
0620 09:00:19.126581 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.126583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.126626 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.126707 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.127095 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45886: BlockingRecv error: recv got EOF from 127.0.0.1:45886 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 09:00:19.127357 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.128391 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.129494 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:00:19.137494 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:19.130237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.1:37857)
0620 09:00:19.130362 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.130365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.130400 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:00:19.130500 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.130502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.130534 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.130591 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.131056 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.131623 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.132360 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.132651 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.132730 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:19.134886 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:19.134889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:19.134891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:19.134911 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:19.136031 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.136428 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.136431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.136432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.136544 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.136759 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.136761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.136762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.136892 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:19.136998 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:19.137179 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:00:19.137300 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 09:00:19.140079 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:19.139187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.98.1:37857)
0620 09:00:19.139300 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.139304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.139407 (+ 103us) server_negotiation.cc:407] Connection header received
0620 09:00:19.139504 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.139507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.139547 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.139626 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.139978 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45894: BlockingRecv error: recv got EOF from 127.0.0.1:45894 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 09:00:19.140538 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781945987698834-392-0/krb5kdc/krb5cc)
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.141503 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:19.142357 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:00:19.150756 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:19.143109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.1:37857)
0620 09:00:19.143225 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:19.143230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:19.143365 (+ 135us) server_negotiation.cc:407] Connection header received
0620 09:00:19.143500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:19.143503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:19.143540 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:19.143637 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:19.144247 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.144815 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.145698 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:19.145897 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:19.145975 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:19.148041 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:19.148044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:19.148047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:19.148067 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:19.149521 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.149865 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.149867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.149869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.149936 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:19.150149 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:19.150151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:19.150152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:19.150304 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:19.150400 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:19.150556 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:00:19.150636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:00:19.151743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6059
I20260620 09:00:19.157948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6193
I20260620 09:00:19.163329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6327
I20260620 09:00:19.168331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 5965
2026-06-20T09:00:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2106 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[6476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[6476](info): set up 2 sockets
Jun 20 09:00:19 dist-test-slave-scsf krb5kdc[6476](info): commencing operation
krb5kdc: starting...
W20260620 09:00:20.196861 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:00:20.316205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:20.428427 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:20.428790 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:20.428869 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:20.433055 6489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:20.433149 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:20.433194 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:20.433252 6489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:20.433293 6489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:20.438833 6489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:20.439976 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:20.441157 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:20.449566 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:20.449599 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:20.449674 6489 server_base.cc:1034] running on GCE node
W20260620 09:00:20.449566 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:20.450429 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:20.451542 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:20.452723 6489 hybrid_clock.cc:648] HybridClock initialized: now 1781946020452708 us; error 25 us; skew 500 ppm
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:20.455382 6489 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:00:20.456316 6489 webserver.cc:460] Webserver started at http://127.0.98.62:37521/ using document root <none> and password file <none>
I20260620 09:00:20.456930 6489 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:20.457023 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:20.457257 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:20.459229 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "23429c57255a498f99daa1ec36d040d8"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "2da2743313f93da8f3d30792b7ae686e"
server_key_iv: "33e54ec3d1f9aa4fefa3127843ca8ddf"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.459777 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "23429c57255a498f99daa1ec36d040d8"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "2da2743313f93da8f3d30792b7ae686e"
server_key_iv: "33e54ec3d1f9aa4fefa3127843ca8ddf"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.463563 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:00:20.466100 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:20.467377 6489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:20.467521 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "23429c57255a498f99daa1ec36d040d8"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "2da2743313f93da8f3d30792b7ae686e"
server_key_iv: "33e54ec3d1f9aa4fefa3127843ca8ddf"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.467638 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:20.500408 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:20.503602 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:20.503823 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:20.511075 6489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36813
I20260620 09:00:20.511382 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36813 every 8 connection(s)
I20260620 09:00:20.512125 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:20.512678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6489
I20260620 09:00:20.512847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:20.513111 392 external_mini_cluster.cc:1428] Setting key 07885e1939d31782d9f92db89d844244
I20260620 09:00:20.515413 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:20.521122 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Bootstrap starting.
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:20.523792 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:20.524595 6557 log.cc:824] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:20.526907 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: No bootstrap required, opened a new log
I20260620 09:00:20.527629 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:20.514557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.0.98.62:36813)
0620 09:00:20.514984 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:20.514999 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:20.515035 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:00:20.515677 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:20.515692 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:20.516014 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:20.516364 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:20.517174 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.518250 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.519128 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.519614 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.519799 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:20.522675 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:20.522680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:20.522693 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:20.522725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:20.524620 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:20.525149 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:20.525155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:20.525161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:20.525326 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:20.525658 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:20.525660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:20.525661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:20.526059 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:20.526218 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:20.526520 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 09:00:20.526833 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":146,"threads_started":1}
I20260620 09:00:20.530965 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } }
I20260620 09:00:20.531186 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:20.531268 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23429c57255a498f99daa1ec36d040d8, State: Initialized, Role: FOLLOWER
I20260620 09:00:20.531687 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } }
I20260620 09:00:20.531850 6557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:20.531913 6557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:20.532051 6557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:20.533123 6557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } }
I20260620 09:00:20.533483 6557 leader_election.cc:302] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23429c57255a498f99daa1ec36d040d8; no voters:
I20260620 09:00:20.533806 6557 leader_election.cc:288] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:20.533946 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:20.534185 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 LEADER]: Becoming Leader. State: Replica: 23429c57255a498f99daa1ec36d040d8, State: Running, Role: LEADER
I20260620 09:00:20.534574 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } }
I20260620 09:00:20.535115 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:20.536181 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23429c57255a498f99daa1ec36d040d8. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } }
I20260620 09:00:20.536326 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:20.536763 6571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:20.536777 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } }
I20260620 09:00:20.537699 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:20.540354 6571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:20.546206 6571 catalog_manager.cc:1347] Generated new cluster ID: 4f93bd4c69cd41f1b8120d9e2c7af65f
I20260620 09:00:20.546281 6571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:20.557873 6571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:20.558854 6571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:20.564082 6571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Generated new TSK 0
I20260620 09:00:20.564862 6571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:00:20.601889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:20.730875 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:20.731151 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:20.731216 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:20.735244 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:20.735307 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:20.735400 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:20.740918 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:20.742060 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:20.743108 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:20.752581 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:20.752575 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:20.752682 6583 server_base.cc:1034] running on GCE node
W20260620 09:00:20.752570 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:20.753278 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:20.753880 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:20.755065 6583 hybrid_clock.cc:648] HybridClock initialized: now 1781946020755030 us; error 55 us; skew 500 ppm
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:20.758174 6583 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:00:20.759047 6583 webserver.cc:460] Webserver started at http://127.0.98.1:46691/ using document root <none> and password file <none>
I20260620 09:00:20.759573 6583 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:20.759630 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:20.759802 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:20.761852 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "1fa651c62fe848f39e269be2fdab4f57"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "0f523c7b1fecf42e6edd81143be28202"
server_key_iv: "b682a55470c3ca8c19194ef0c291c16b"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.762358 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1fa651c62fe848f39e269be2fdab4f57"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "0f523c7b1fecf42e6edd81143be28202"
server_key_iv: "b682a55470c3ca8c19194ef0c291c16b"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.765981 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:00:20.768337 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:20.769601 6583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:20.769711 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "1fa651c62fe848f39e269be2fdab4f57"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "0f523c7b1fecf42e6edd81143be28202"
server_key_iv: "b682a55470c3ca8c19194ef0c291c16b"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.769796 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:20.788800 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:20.792148 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:20.792305 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:20.792908 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:20.793691 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:20.793737 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:20.793777 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:20.793799 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:20.802340 6583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40363
I20260620 09:00:20.802362 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40363 every 8 connection(s)
I20260620 09:00:20.803247 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:20.809820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6583
I20260620 09:00:20.809959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:20.810267 392 external_mini_cluster.cc:1428] Setting key 2578165135c6de0444f7ab3e11c8a828
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:20.818274 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:20.805116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60797 (local address 127.0.98.62:36813)
0620 09:00:20.805260 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:20.805263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:20.806097 (+ 834us) server_negotiation.cc:407] Connection header received
0620 09:00:20.807489 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:20.807492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:20.807552 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:20.807650 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:20.808886 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.809458 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.811035 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.811287 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.811403 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:20.814527 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:20.814531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:20.814533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:20.814557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:20.816079 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:20.816703 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:20.816706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:20.816708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:20.816789 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:20.817135 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:20.817138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:20.817139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:20.817322 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:20.817425 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:20.817963 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 09:00:20.818090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:00:20.819090 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:20.805433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:60797)
0620 09:00:20.805923 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:20.805964 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:20.807257 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:20.807825 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:20.807831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:20.808190 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:20.808734 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:20.808749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.809617 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.809622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:20.810889 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:20.810899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:20.811456 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:20.811460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:20.811717 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:20.812337 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:20.812362 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:20.814320 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:20.816230 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:20.816236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:20.816252 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:20.816587 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:20.816882 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:20.816885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:20.816887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:20.817035 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:20.817423 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:20.817430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:20.817717 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 09:00:20.817906 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 09:00:20.818246 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:20.820328 6712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:20.820616 6712 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:20.821245 6712 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:20.823005 6521 ts_manager.cc:194] Registered new tserver with Master: 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363)
I20260620 09:00:20.824117 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 09:00:20.839361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:20.950477 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:20.950800 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:20.950876 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:20.955236 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:20.955322 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:20.955415 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:20.960912 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:20.962065 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:20.963160 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:20.973088 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:20.973070 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:20.973070 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:20.973433 6717 server_base.cc:1034] running on GCE node
I20260620 09:00:20.973975 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:20.974735 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:20.975922 6717 hybrid_clock.cc:648] HybridClock initialized: now 1781946020975895 us; error 67 us; skew 500 ppm
Jun 20 09:00:20 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:20.978771 6717 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:00:20.979667 6717 webserver.cc:460] Webserver started at http://127.0.98.2:38815/ using document root <none> and password file <none>
I20260620 09:00:20.980253 6717 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:20.980346 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:20.980635 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:20.982709 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d762d9cd479416da10b933a0a94eacf"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "14aea3160df9ccdf39385548f89fa826"
server_key_iv: "9b9be922c76f8a7636bccbc853bbfcbd"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.983300 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d762d9cd479416da10b933a0a94eacf"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "14aea3160df9ccdf39385548f89fa826"
server_key_iv: "9b9be922c76f8a7636bccbc853bbfcbd"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.987221 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:00:20.989727 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:20.990929 6717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:20.991070 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "4d762d9cd479416da10b933a0a94eacf"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "14aea3160df9ccdf39385548f89fa826"
server_key_iv: "9b9be922c76f8a7636bccbc853bbfcbd"
server_key_version: "encryptionkey@0"
I20260620 09:00:20.991182 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.017251 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:21.020709 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:21.020903 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:21.021533 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:21.022387 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:21.022460 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.022533 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:21.022578 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.032136 6717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36283
I20260620 09:00:21.032153 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36283 every 8 connection(s)
I20260620 09:00:21.033145 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:21.036149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6717
I20260620 09:00:21.036259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:21.036608 392 external_mini_cluster.cc:1428] Setting key 3e84893c27d3e6f513127f62d2b5820c
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:21.048804 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.034981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47089 (local address 127.0.98.62:36813)
0620 09:00:21.035167 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.035172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.036165 (+ 993us) server_negotiation.cc:407] Connection header received
0620 09:00:21.037865 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.037871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.037925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.037993 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.039194 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.039902 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.041597 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.041816 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.041904 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.044982 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.044985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.044987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.045011 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.046518 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.047044 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.047047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.047049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.047130 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.047560 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.047563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.047565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.047760 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.047877 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.048458 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.048584 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:21.049782 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.035471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:47089)
0620 09:00:21.036006 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.036063 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.037600 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.038202 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.038210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.038566 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.039045 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.039058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.040033 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.040037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.041466 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.041480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.041992 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.041998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.042259 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.042952 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.042980 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.044800 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.046641 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.046646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.046663 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.046927 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.047241 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.047244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.047246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.047438 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.047886 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.047895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.048154 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.048388 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.048745 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260620 09:00:21.051221 6846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:21.051478 6846 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:21.051985 6846 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
I20260620 09:00:21.053148 6521 ts_manager.cc:194] Registered new tserver with Master: 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
I20260620 09:00:21.053730 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:00:21.064899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:21.177556 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:21.177829 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:21.177872 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:21.181890 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:21.181950 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:21.182011 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:21.187340 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:21.188419 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:21.189481 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:21.199029 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.199074 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.199239 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:21.199121 6851 server_base.cc:1034] running on GCE node
I20260620 09:00:21.199728 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:21.200289 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:21.201501 6851 hybrid_clock.cc:648] HybridClock initialized: now 1781946021201483 us; error 41 us; skew 500 ppm
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:21.204367 6851 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:00:21.205299 6851 webserver.cc:460] Webserver started at http://127.0.98.3:37079/ using document root <none> and password file <none>
I20260620 09:00:21.205837 6851 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.205895 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.206056 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:21.208106 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f05f49e630b472db248ebaa113915f1"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "84a9b3432996edff942a3872a2ecd566"
server_key_iv: "88c637ad73c59d3b35d873e3a64fd72b"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.208690 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f05f49e630b472db248ebaa113915f1"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "84a9b3432996edff942a3872a2ecd566"
server_key_iv: "88c637ad73c59d3b35d873e3a64fd72b"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.212410 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:00:21.214811 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.216001 6851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:21.216127 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "3f05f49e630b472db248ebaa113915f1"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "84a9b3432996edff942a3872a2ecd566"
server_key_iv: "88c637ad73c59d3b35d873e3a64fd72b"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.216241 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.241669 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:21.244745 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:21.244953 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:21.245618 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:21.246474 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:21.246523 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.246564 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:21.246579 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.255620 6851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40699
I20260620 09:00:21.255643 6979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40699 every 8 connection(s)
I20260620 09:00:21.256624 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:21.261538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6851
I20260620 09:00:21.261698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:21.261974 392 external_mini_cluster.cc:1428] Setting key ae83996903bcc7d5be00125888c6ff4c
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:21.270365 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.258554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38715 (local address 127.0.98.62:36813)
0620 09:00:21.258706 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.258710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.259511 (+ 801us) server_negotiation.cc:407] Connection header received
0620 09:00:21.260891 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.260894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.260941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.261022 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.262329 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.263033 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.264396 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.264671 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.264764 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.267113 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.267117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.267120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.267145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.268579 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.269040 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.269043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.269044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.269117 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.269398 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.269399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.269400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.269564 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.269646 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.270122 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.270244 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:00:21.271337 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.258897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:38715)
0620 09:00:21.259349 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.259393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.260686 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.261196 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.261203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.261562 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.262170 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.262185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.263228 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.263231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.264287 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.264297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.264821 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.264824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.264975 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.265514 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.265530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.266953 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.268689 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.268693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.268708 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.268957 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.269194 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.269196 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.269197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.269323 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.269669 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.269674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.269912 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.270070 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.270357 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260620 09:00:21.272755 6980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:21.273018 6980 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:21.273567 6980 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
I20260620 09:00:21.274605 6521 ts_manager.cc:194] Registered new tserver with Master: 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:21.275390 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38715
I20260620 09:00:21.275889 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 09:00:21.303077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34855
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36813,127.0.98.61:34855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:21.417603 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:21.417898 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:21.417984 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:21.422242 6985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:21.422340 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:21.422387 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:21.422444 6985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:21.422495 6985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:21.427778 6985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36813,127.0.98.61:34855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
W20260620 09:00:21.428030 6985 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36813,127.0.98.61:34855'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 09:00:21.428984 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:21.430032 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:21.438648 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.438781 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.438695 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:21.438681 6985 server_base.cc:1034] running on GCE node
I20260620 09:00:21.439440 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:21.440624 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:21.441820 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781946021441806 us; error 25 us; skew 500 ppm
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:21.444540 6985 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:00:21.445370 6985 webserver.cc:460] Webserver started at http://127.0.98.61:39797/ using document root <none> and password file <none>
I20260620 09:00:21.445894 6985 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.445948 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.446135 6985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:21.448232 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/instance:
uuid: "450e1074deee4223aab2b99b333566b8"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "438ad7331da07d158c669d8af4703ea3"
server_key_iv: "cb7d6ab68ba5bad5ba81316262fc83fd"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.448810 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal/instance:
uuid: "450e1074deee4223aab2b99b333566b8"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "438ad7331da07d158c669d8af4703ea3"
server_key_iv: "cb7d6ab68ba5bad5ba81316262fc83fd"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.452415 6985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:21.454869 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.456094 6985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:21.456250 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "450e1074deee4223aab2b99b333566b8"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "438ad7331da07d158c669d8af4703ea3"
server_key_iv: "cb7d6ab68ba5bad5ba81316262fc83fd"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.456358 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.482235 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:21.485296 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:21.485530 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:21.492499 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34855
I20260620 09:00:21.492777 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34855 every 8 connection(s)
I20260620 09:00:21.493541 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:00:21.496366 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:21.500073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 6985
I20260620 09:00:21.500371 7053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:00:21.513705 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.502404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.0.98.62:36813)
0620 09:00:21.502526 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.502531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.502851 (+ 320us) server_negotiation.cc:407] Connection header received
0620 09:00:21.503334 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.503337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.503392 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.503498 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.504536 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.505061 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.506199 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.506437 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.506519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.509486 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.509490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.509493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.509523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.511135 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.511721 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.511725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.511727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.511828 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.512602 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.512605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.512608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.512867 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.512940 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.513419 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.513548 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:00:21.514693 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.501747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.0.98.61:34855)
0620 09:00:21.502172 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.502193 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.502231 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:00:21.503129 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.503148 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.503348 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.503521 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.504315 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.505427 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.506251 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.506665 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.506759 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.509878 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.509887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.509903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.509929 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.511823 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.512289 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.512293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.512294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.512403 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.512754 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.512758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.512763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.513125 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.513341 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.513563 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.513865 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260620 09:00:21.514695 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.502291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47102)
0620 09:00:21.502718 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.502758 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.503142 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.503567 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.503572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.504003 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.504307 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.504319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.505207 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.505211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.506071 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.506079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.506577 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.506580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.506753 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.507756 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.507771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.509324 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.511231 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.511239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.511251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.511607 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.512301 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.512304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.512307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.512488 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.513023 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.513029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.513174 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.514113 (+ 939us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.514294 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":55,"thread_start_us":128,"threads_started":1}
I20260620 09:00:21.516103 7053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:00:21.524230 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.516653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46892)
0620 09:00:21.516888 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.516903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.516989 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.517204 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.517206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.517372 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.517447 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.517451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.518134 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.518137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.518645 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.518652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.519081 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.519084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.519213 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.519696 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.519707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.521117 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.522710 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.522713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.522716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.522962 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.523228 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.523231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.523233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.523319 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.523700 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.523702 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.523799 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.523896 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.524038 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 09:00:21.524230 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.516780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.0.98.61:34855)
0620 09:00:21.516891 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.516895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.516950 (+ 55us) server_negotiation.cc:407] Connection header received
0620 09:00:21.517073 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.517076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.517118 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.517197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.517548 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.518032 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.518747 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.518984 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.519107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.521255 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.521258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.521260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.521280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.522601 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.523058 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.523060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.523062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.523142 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.523430 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.523433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.523434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.523611 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.523695 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.523958 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.524095 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":38}
I20260620 09:00:21.527336 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Bootstrap starting.
I20260620 09:00:21.529734 7053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:21.530508 7053 log.cc:824] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:21.532517 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: No bootstrap required, opened a new log
I20260620 09:00:21.535725 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } }
I20260620 09:00:21.535970 7053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:21.536052 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450e1074deee4223aab2b99b333566b8, State: Initialized, Role: FOLLOWER
I20260620 09:00:21.536571 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } }
I20260620 09:00:21.537096 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } } }
I20260620 09:00:21.537278 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:21.537856 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:00:21.542637 7070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:21.542774 7070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:21.544411 6985 master_runner.cc:428] Detected that this master 450e1074deee4223aab2b99b333566b8 is joining an existing cluster
I20260620 09:00:21.544519 6985 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34855
I20260620 09:00:21.554704 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.546489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47116)
0620 09:00:21.546836 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.546852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.546981 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.547239 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.547242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.547457 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.547557 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.547564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.548589 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.548592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.549242 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.549250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.549724 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.549727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.549811 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.550389 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.550406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.550875 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.552982 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.552986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.552988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.553256 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.553618 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.553626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.553709 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.554168 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.554170 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.554244 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.554352 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.554493 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260620 09:00:21.554811 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.546543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.98.62:36813)
0620 09:00:21.546669 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.546674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.546938 (+ 264us) server_negotiation.cc:407] Connection header received
0620 09:00:21.547086 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.547090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.547143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.547230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.547673 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.548470 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.549360 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.549585 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.549718 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.551023 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.551026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.551029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.551052 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.552855 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553415 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.553420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.553422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.553503 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553780 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.553784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.553786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.554038 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.554129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.554402 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.554647 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:21.554905 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.546635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.0.98.61:34855)
0620 09:00:21.546821 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.546825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.547150 (+ 325us) server_negotiation.cc:407] Connection header received
0620 09:00:21.547283 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.547286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.547322 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.547369 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.547946 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.548645 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.549792 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.550112 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.550191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.551489 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.551494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.551501 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.551521 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.553255 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553714 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.553719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.553720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.553821 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.554099 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.554102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.554104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.554306 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.554396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.554649 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.554758 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 09:00:21.555209 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.546721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46894)
0620 09:00:21.547051 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.547065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.547173 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.547490 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.547493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.547710 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.547828 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.547835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.548749 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.548753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.549654 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.549665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.550344 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.550348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.550438 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.550949 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.550964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.551371 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.553360 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.553370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.553608 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.553924 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.553926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.553928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.554003 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.554430 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.554433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.554501 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.554949 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.555061 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:00:21.569027 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.559850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.0.98.61:34855)
0620 09:00:21.559959 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.559965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.560311 (+ 346us) server_negotiation.cc:407] Connection header received
0620 09:00:21.560606 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.560610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.560660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.560752 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.561424 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.561917 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.562790 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.563021 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.563127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.565430 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.565434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.565435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.565456 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.566868 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.567357 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.567364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.567366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.567444 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.567980 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.567983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.567985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.568194 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.568281 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.568749 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.568865 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:00:21.569038 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.559826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46910)
0620 09:00:21.560168 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.560241 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.560430 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.560786 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.560791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.561206 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.561310 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.561316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.562024 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.562028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.562664 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.562672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.563132 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.563135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.563245 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.563791 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.563803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.565283 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.566995 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.567003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.567010 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.567256 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.567737 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.567740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.567742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.567833 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.568340 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.568347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.568517 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.568681 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.568885 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260620 09:00:21.570849 6520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34855: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "450e1074deee4223aab2b99b333566b8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34855
}
attrs {
promote: true
}
}
dest_uuid: "23429c57255a498f99daa1ec36d040d8"
cas_config_opid_index: -1
I20260620 09:00:21.571586 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } }
I20260620 09:00:21.573706 7081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 450e1074deee4223aab2b99b333566b8 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } }
I20260620 09:00:21.574497 7027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:21.575481 7081 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34855
I20260620 09:00:21.575887 6985 master.cc:537] Master@127.0.98.61:34855 shutting down...
I20260620 09:00:21.575944 7027 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 09:00:21.575914 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } }
I20260620 09:00:21.576026 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:21.576833 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260620 09:00:21.578913 6507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 09:00:21.580191 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 -> Peer 450e1074deee4223aab2b99b333566b8 (127.0.98.61:34855): Couldn't send request to peer 450e1074deee4223aab2b99b333566b8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 09:00:21.580880 6985 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:21.581169 6985 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:21.581303 6985 tablet_replica.cc:330] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: stopping tablet replica
I20260620 09:00:21.586164 6985 master.cc:552] Master@127.0.98.61:34855 shutdown complete.
I20260620 09:00:21.586230 6985 master_runner.cc:305] Clearing existing system tablet
I20260620 09:00:21.586480 6985 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.586536 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.587842 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:00:21.588759 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.589025 6985 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.589291 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:21.589360 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "450e1074deee4223aab2b99b333566b8"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "438ad7331da07d158c669d8af4703ea3"
server_key_iv: "cb7d6ab68ba5bad5ba81316262fc83fd"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.591498 6985 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:21.594178 6985 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:00:21.594344 6985 log.cc:1196] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 09:00:21.594589 6985 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Deleting consensus metadata
I20260620 09:00:21.594764 6985 master_runner.cc:315] Copying system tablet from 127.0.98.62:36813
I20260620 09:00:21.595922 6985 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 09:00:21.598018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43387
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:00:21.605125 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.596699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.0.98.62:36813)
0620 09:00:21.596847 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.596852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.596935 (+ 83us) server_negotiation.cc:407] Connection header received
0620 09:00:21.597116 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.597119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.597171 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.597287 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.597806 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.598638 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.599759 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.599985 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.600097 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.601851 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.601854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.601857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.601888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.603367 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.603812 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.603815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.603817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.603897 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.604174 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.604178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.604180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.604366 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.604511 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.604843 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.604952 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:00:21.605139 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.596599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47130)
0620 09:00:21.596848 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.596864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.596992 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.597304 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.597307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.597538 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.597656 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.597663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.598765 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.598772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.599631 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.599640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.600100 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.600104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.600187 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.601205 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.601217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.601654 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.603494 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.603498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.603500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.603711 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.603997 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.604000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.604002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.604086 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.604544 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.604547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.604641 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.604787 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.604948 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260620 09:00:21.605983 6541 tablet_copy_service.cc:140] P 23429c57255a498f99daa1ec36d040d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 450e1074deee4223aab2b99b333566b8 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47130)
I20260620 09:00:21.606168 6541 tablet_copy_service.cc:161] P 23429c57255a498f99daa1ec36d040d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 450e1074deee4223aab2b99b333566b8 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47130: session id = 450e1074deee4223aab2b99b333566b8-00000000000000000000000000000000
I20260620 09:00:21.607563 6541 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:00:21.609094 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:00:21.610926 6985 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: tablet copy: Starting download of 0 data blocks...
I20260620 09:00:21.611125 6985 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: tablet copy: Starting download of 1 WAL segments...
I20260620 09:00:21.612821 6985 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:00:21.614362 6541 tablet_copy_service.cc:342] P 23429c57255a498f99daa1ec36d040d8: Request end of tablet copy session 450e1074deee4223aab2b99b333566b8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47130
I20260620 09:00:21.614485 6541 tablet_copy_service.cc:433] P 23429c57255a498f99daa1ec36d040d8: ending tablet copy session 450e1074deee4223aab2b99b333566b8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 450e1074deee4223aab2b99b333566b8
I20260620 09:00:21.621956 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:21.623672 7095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.623747 7098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:21.624119 6985 server_base.cc:1034] running on GCE node
W20260620 09:00:21.628634 7096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:21.628901 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:21.629267 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:21.630417 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781946021630405 us; error 35 us; skew 500 ppm
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:21.631942 6985 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:00:21.632524 6985 webserver.cc:460] Webserver started at http://127.0.98.61:41745/ using document root <none> and password file <none>
I20260620 09:00:21.632726 6985 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.632800 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.633932 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:21.635309 7105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.635536 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:21.635632 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "450e1074deee4223aab2b99b333566b8"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "438ad7331da07d158c669d8af4703ea3"
server_key_iv: "cb7d6ab68ba5bad5ba81316262fc83fd"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.635740 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.660851 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:21.663017 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:21.667963 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34855
I20260620 09:00:21.668291 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34855 every 8 connection(s)
I20260620 09:00:21.668751 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:00:21.670146 7158 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:00:21.671052 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Bootstrap starting.
I20260620 09:00:21.679564 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:00:21.679911 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: Bootstrap complete.
I20260620 09:00:21.680423 7158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } }
I20260620 09:00:21.680612 7158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 450e1074deee4223aab2b99b333566b8, State: Initialized, Role: LEARNER
I20260620 09:00:21.680744 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } }
I20260620 09:00:21.681129 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } }
I20260620 09:00:21.681298 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:21.681298 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:00:21.684401 7172 catalog_manager.cc:1259] Loaded cluster ID: 4f93bd4c69cd41f1b8120d9e2c7af65f
I20260620 09:00:21.684540 7172 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: loading cluster ID for follower catalog manager: success
I20260620 09:00:21.686050 7172 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: acquiring CA information for follower catalog manager: success
I20260620 09:00:21.687183 7172 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 09:00:21.755155 7094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:21.755447 7094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:21.755489 7094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:21.759464 7094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:21.759531 7094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:21.759550 7094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:21.759568 7094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:21.759583 7094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:21.764953 7094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:21.765964 7094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:21.766940 7094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:21.775224 7177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.775264 7178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:21.775224 7180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:21.775498 7094 server_base.cc:1034] running on GCE node
I20260620 09:00:21.776103 7094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:21.777344 7094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:21.778524 7094 hybrid_clock.cc:648] HybridClock initialized: now 1781946021778491 us; error 48 us; skew 500 ppm
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:21.781258 7094 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:00:21.782147 7094 webserver.cc:460] Webserver started at http://127.0.98.60:32807/ using document root <none> and password file <none>
I20260620 09:00:21.782737 7094 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.782852 7094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.783092 7094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:21.785295 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/instance:
uuid: "060d38baef5d4da78af1530433048e73"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "4283f255b5940f9be078173837655a94"
server_key_iv: "fd5bf6d2da7e60166c94686a51a23972"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.785871 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal/instance:
uuid: "060d38baef5d4da78af1530433048e73"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "4283f255b5940f9be078173837655a94"
server_key_iv: "fd5bf6d2da7e60166c94686a51a23972"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.789683 7094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 09:00:21.792240 7187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.793524 7094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:21.793665 7094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "060d38baef5d4da78af1530433048e73"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "4283f255b5940f9be078173837655a94"
server_key_iv: "fd5bf6d2da7e60166c94686a51a23972"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.793785 7094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.806985 7094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:21.809897 7094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:21.810048 7094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:21.816870 7094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43387
I20260620 09:00:21.817147 7239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43387 every 8 connection(s)
I20260620 09:00:21.817888 7094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:00:21.821065 7240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:21.824977 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:00:21.825889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 7094
I20260620 09:00:21.827085 6712 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:00:21.840696 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.827452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.98.62:36813)
0620 09:00:21.827614 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.827618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.828298 (+ 680us) server_negotiation.cc:407] Connection header received
0620 09:00:21.829067 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.829070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.829122 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.829217 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.830320 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.831129 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.832611 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.832946 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.833057 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.836610 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.836616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.836620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.836656 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.838323 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.838874 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.838877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.838878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.838951 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.839340 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.839346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.839350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.839602 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.839707 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.840368 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.840518 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 09:00:21.841679 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.827503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47140)
0620 09:00:21.828149 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.828200 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.828866 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.829302 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.829307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.829784 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.830159 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.830189 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.831263 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.831265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.832412 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.832424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.833094 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.833096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.833262 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.833979 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.833999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.836458 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.838439 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.838445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.838457 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.838769 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.839041 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.839046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.839051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.839228 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.839727 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.839733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.840089 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.840286 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.840693 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":2,"spinlock_wait_cycles":118400,"thread_start_us":207,"threads_started":1}
I20260620 09:00:21.841677 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.827506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.0.98.60:43387)
0620 09:00:21.827982 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.827999 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.828038 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:00:21.828849 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.828868 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.829105 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.829276 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.830175 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.831403 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.832540 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.833137 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.833260 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.836695 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.836701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.836711 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.836731 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.838947 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.839383 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.839388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.839389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.839454 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.839726 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.839729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.839731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.840083 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.840283 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.840530 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.840832 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":253,"server-negotiator.queue_time_us":345,"thread_start_us":172,"threads_started":1}
I20260620 09:00:21.842933 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:00:21.851552 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.843416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46924)
0620 09:00:21.843831 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.843848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.843920 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.844131 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.844134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.844268 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.844362 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.844367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.845300 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.845302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.845844 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.845851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.846267 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.846269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.846340 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.846764 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.846775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.847955 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.849736 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.849738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.849740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.849969 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.850369 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.850372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.850374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.850470 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.850983 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.850986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.851085 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.851190 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.851368 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340}
I20260620 09:00:21.851722 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.843547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.0.98.61:34855)
0620 09:00:21.843872 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.843885 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.843907 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:00:21.843997 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.843999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.844040 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.844107 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.844507 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.845210 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.845937 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.846160 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.846254 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.848082 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.848086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.848088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.848112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.849638 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.850113 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.850116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.850118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.850271 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.850597 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.850601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.850602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.850837 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.850937 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.851324 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.851499 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":130,"threads_started":1}
I20260620 09:00:21.852608 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:00:21.860209 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.853147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55912)
0620 09:00:21.853398 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.853407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.853476 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.853642 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.853644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.853774 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.853846 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.853850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.854507 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.854509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.854999 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.855005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.855454 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.855456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.855513 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.855920 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.855931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.857203 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.858962 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.858964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.858966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.859150 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.859383 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.859385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.859386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.859456 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.859769 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.859771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.859844 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.859922 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.860035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260620 09:00:21.860224 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.853222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.0.98.60:43387)
0620 09:00:21.853352 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.853356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.853469 (+ 113us) server_negotiation.cc:407] Connection header received
0620 09:00:21.853534 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.853536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.853571 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.853628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.853931 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.854421 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.855093 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.855358 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.855437 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.857339 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.857342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.857344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.857364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.858858 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.859234 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.859237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.859238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.859305 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.859528 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.859530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.859531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.859693 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.859764 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.859973 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.860091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":56}
I20260620 09:00:21.863559 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Bootstrap starting.
I20260620 09:00:21.866024 7240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:21.866786 7240 log.cc:824] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:21.868816 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: No bootstrap required, opened a new log
I20260620 09:00:21.871896 7240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } }
I20260620 09:00:21.872107 7240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:21.872180 7240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d38baef5d4da78af1530433048e73, State: Initialized, Role: FOLLOWER
I20260620 09:00:21.872668 7240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } }
I20260620 09:00:21.873238 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } } }
I20260620 09:00:21.873353 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:21.874056 7240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:00:21.878878 7257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:21.879014 7257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:21.882503 7094 master_runner.cc:428] Detected that this master 060d38baef5d4da78af1530433048e73 is joining an existing cluster
I20260620 09:00:21.882592 7094 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43387
I20260620 09:00:21.894956 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.885917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55916)
0620 09:00:21.886604 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.886616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.886717 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.887020 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.887022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.887242 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.887408 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.887418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888369 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.888372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.889905 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.889915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890553 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.890643 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.891189 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.891200 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.891548 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.893446 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.893448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.893450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.893642 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.893906 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.893908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.893909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.893978 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.894386 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.894388 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.894489 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.894590 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.894733 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":49,"thread_start_us":87,"threads_started":1}
I20260620 09:00:21.894989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6583
I20260620 09:00:21.895126 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.886116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.0.98.60:43387)
0620 09:00:21.886221 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.886225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.886664 (+ 439us) server_negotiation.cc:407] Connection header received
0620 09:00:21.886801 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.886803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.886840 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.886888 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.887504 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888221 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890028 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890333 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890400 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.891679 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.891683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.891686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.891709 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.893329 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.893734 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.893738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.893740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.893821 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.894083 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.894088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.894090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.894294 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.894395 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.894643 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.894936 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":43}
I20260620 09:00:21.903540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40363
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36813
--webserver_port=46691
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:00:21.913645 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.885528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.0.98.61:34855)
0620 09:00:21.885676 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.885681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.886381 (+ 700us) server_negotiation.cc:407] Connection header received
0620 09:00:21.886503 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.886506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.886558 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.886628 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.887508 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888307 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890263 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890613 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.909942 (+ 19250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.909947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.909950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.909985 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.911600 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912232 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.912236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.912238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.912323 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912639 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.912643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.912645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.912894 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.912964 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.913331 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.913468 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:00:21.914196 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.885663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.98.62:36813)
0620 09:00:21.885800 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.885806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.886260 (+ 454us) server_negotiation.cc:407] Connection header received
0620 09:00:21.886459 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.886464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.886524 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.886598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.887440 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888228 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890138 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890460 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890538 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.910023 (+ 19485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.910031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.910036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.910075 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.911894 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912439 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.912445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.912449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.912578 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912971 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.912975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.912977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.913240 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.913333 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.913766 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.913912 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:00:21.914227 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.885899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46934)
0620 09:00:21.886289 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.886303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.886416 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.886741 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.886744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.887142 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.887372 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.887380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888447 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.888450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.890117 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.890127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890848 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890866 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.890960 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.909075 (+ 18115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.909094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.909716 (+ 622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.911729 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.911731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.911733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.911978 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.912430 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.912435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.912541 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.913032 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.913035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.913141 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.913867 (+ 726us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.914019 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":671,"spinlock_wait_cycles":1280,"thread_start_us":90,"threads_started":1}
I20260620 09:00:21.915212 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.885475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47142)
0620 09:00:21.886164 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.886190 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.886332 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.886737 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.886743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.887116 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.887305 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.887312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.888369 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.888372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.890031 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.890038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.890673 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.890676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.890787 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.909136 (+ 18349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.909158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.909794 (+ 636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.912039 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.912046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.912304 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.912711 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.912717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.912722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.912848 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.913418 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.913422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.913535 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.914759 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.914941 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":713,"spinlock_wait_cycles":1280,"thread_start_us":122,"threads_started":1}
Jun 20 09:00:21 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:00:21.928318 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.919034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55922)
0620 09:00:21.919163 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.919185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.919260 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.919657 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.919661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.919849 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.919949 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.919956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.921009 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.921013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.921899 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.921909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.922529 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.922532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.922642 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.923158 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.923201 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.924842 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.926725 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.926739 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.926741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.926990 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.927311 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.927314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.927316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.927405 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.927801 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.927804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.927900 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.928014 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.928156 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 09:00:21.928342 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.919302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.60:43387)
0620 09:00:21.919401 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.919405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.919417 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:00:21.919484 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.919486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.919533 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.919611 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.920071 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.920894 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.922017 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.922405 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.922527 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.924986 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.924990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.924992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.925017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.926606 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.927091 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.927094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.927096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.927198 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.927494 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.927497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.927498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.927700 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.927795 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.928072 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.928194 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:00:21.929818 6520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "060d38baef5d4da78af1530433048e73"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43387
}
attrs {
promote: true
}
}
dest_uuid: "23429c57255a498f99daa1ec36d040d8"
cas_config_opid_index: 5
I20260620 09:00:21.930425 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } }
I20260620 09:00:21.931559 7081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 060d38baef5d4da78af1530433048e73 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.932981 7081 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43387
I20260620 09:00:21.933147 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.933245 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:21.934221 7214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:21.935633 7214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:00:21.936117 7094 master.cc:537] Master@127.0.98.60:43387 shutting down...
I20260620 09:00:21.936300 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:00:21.939105 6505 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 -> Peer 060d38baef5d4da78af1530433048e73 (127.0.98.60:43387): Couldn't send request to peer 060d38baef5d4da78af1530433048e73. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:00:21.942224 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.933492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.61:34855)
0620 09:00:21.933616 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.933621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.937652 (+ 4031us) server_negotiation.cc:407] Connection header received
0620 09:00:21.937865 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.937871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.937945 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.938019 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:21.938421 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.939169 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.940640 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.941311 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.941390 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.941535 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.942010 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.942093 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:21.942618 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.931632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46948)
0620 09:00:21.937569 (+ 5937us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.937587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.937738 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.938191 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.938194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.938210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:21.938301 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.938307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.939313 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.939316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.940502 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.940513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.941747 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.941751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.941843 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.941892 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.942433 (+ 541us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.942512 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5848}
I20260620 09:00:21.943238 7132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 LEARNER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:00:21.943584 7094 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:21.943784 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 09:00:21.943871 7094 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:21.943961 7094 tablet_replica.cc:330] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: stopping tablet replica
I20260620 09:00:21.945860 7132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 060d38baef5d4da78af1530433048e73 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.947786 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.948053 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:00:21.948225 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23429c57255a498f99daa1ec36d040d8. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: true } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.948356 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:00:21.961699 7094 master.cc:552] Master@127.0.98.60:43387 shutdown complete.
I20260620 09:00:21.961766 7094 master_runner.cc:305] Clearing existing system tablet
I20260620 09:00:21.962009 7094 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:21.962054 7094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:21.962092 7082 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: attempting to promote NON_VOTER 450e1074deee4223aab2b99b333566b8 to VOTER
I20260620 09:00:21.962608 7082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } }
I20260620 09:00:21.963500 7094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:21.964370 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:21.964641 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 LEARNER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 09:00:21.964650 7094 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:21.964754 7094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:21.964785 7094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "060d38baef5d4da78af1530433048e73"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "4283f255b5940f9be078173837655a94"
server_key_iv: "fd5bf6d2da7e60166c94686a51a23972"
server_key_version: "encryptionkey@0"
I20260620 09:00:21.965060 7081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 09:00:21.966318 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.963607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55924)
0620 09:00:21.966148 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.966233 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2447}
I20260620 09:00:21.966813 7094 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:21.967183 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 450e1074deee4223aab2b99b333566b8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.969066 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 450e1074deee4223aab2b99b333566b8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.969686 7094 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:00:21.969811 7094 log.cc:1196] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 09:00:21.970118 7094 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Deleting consensus metadata
I20260620 09:00:21.970557 7094 master_runner.cc:315] Copying system tablet from 127.0.98.62:36813
I20260620 09:00:21.970736 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.970867 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:21.971266 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.971378 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:21.971542 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:21.971625 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:21.971859 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.966791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55934)
0620 09:00:21.971667 (+ 4876us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.971748 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4790,"thread_start_us":87,"threads_started":1}
W20260620 09:00:21.971988 6505 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 -> Peer 060d38baef5d4da78af1530433048e73 (127.0.98.60:43387): Couldn't send request to peer 060d38baef5d4da78af1530433048e73. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:00:21.972116 7094 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36813
I20260620 09:00:21.980471 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.972743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.98.62:36813)
0620 09:00:21.972857 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:21.972862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:21.973063 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:00:21.973231 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:21.973234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:21.973278 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:21.973377 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:21.973840 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.974442 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.975537 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.975890 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.975997 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.977171 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:21.977175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:21.977178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:21.977205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:21.978800 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.979231 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.979236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.979238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.979322 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:21.979609 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:21.979612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:21.979614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:21.979803 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:21.979911 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:21.980173 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:00:21.980281 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:21.981063 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:21.972709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47156)
0620 09:00:21.972971 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:21.972983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:21.973113 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:21.973393 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:21.973396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:21.973613 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:21.973722 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.973727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.974565 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.974571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.975417 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:21.975425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:21.976000 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:21.976003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:21.976091 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:21.976692 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:21.976705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:21.977022 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:21.978908 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.978915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.978917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.979123 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.979413 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:21.979418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:21.979420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:21.979506 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:21.979900 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:21.979903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:21.980009 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 09:00:21.980118 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:00:21.980276 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260620 09:00:21.981576 6541 tablet_copy_service.cc:140] P 23429c57255a498f99daa1ec36d040d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 060d38baef5d4da78af1530433048e73 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47156)
I20260620 09:00:21.981679 6541 tablet_copy_service.cc:161] P 23429c57255a498f99daa1ec36d040d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 060d38baef5d4da78af1530433048e73 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47156: session id = 060d38baef5d4da78af1530433048e73-00000000000000000000000000000000
I20260620 09:00:21.982807 6541 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:00:21.984134 7094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:00:21.986552 7094 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: tablet copy: Starting download of 0 data blocks...
I20260620 09:00:21.986752 7094 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: tablet copy: Starting download of 1 WAL segments...
I20260620 09:00:21.988385 7094 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:00:21.990384 6541 tablet_copy_service.cc:342] P 23429c57255a498f99daa1ec36d040d8: Request end of tablet copy session 060d38baef5d4da78af1530433048e73-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47156
I20260620 09:00:21.990476 6541 tablet_copy_service.cc:433] P 23429c57255a498f99daa1ec36d040d8: ending tablet copy session 060d38baef5d4da78af1530433048e73-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 060d38baef5d4da78af1530433048e73
I20260620 09:00:21.998212 7094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:21.999833 7279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.001348 7280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.001745 7094 server_base.cc:1034] running on GCE node
W20260620 09:00:22.001910 7282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.002151 7094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:22.002496 7094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:22.003647 7094 hybrid_clock.cc:648] HybridClock initialized: now 1781946022003636 us; error 31 us; skew 500 ppm
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:22.005445 7094 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:00:22.005995 7094 webserver.cc:460] Webserver started at http://127.0.98.60:34261/ using document root <none> and password file <none>
I20260620 09:00:22.006276 7094 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:22.006382 7094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:22.007779 7094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:22.008930 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.009235 7094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:22.009307 7094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "060d38baef5d4da78af1530433048e73"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "4283f255b5940f9be078173837655a94"
server_key_iv: "fd5bf6d2da7e60166c94686a51a23972"
server_key_version: "encryptionkey@0"
I20260620 09:00:22.009384 7094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:22.026270 7094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:22.028723 7094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:22.033691 7094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43387
I20260620 09:00:22.034293 7094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:00:22.035212 7341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43387 every 8 connection(s)
I20260620 09:00:22.037111 7342 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:00:22.038242 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Bootstrap starting.
I20260620 09:00:22.050420 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:00:22.050760 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Bootstrap complete.
I20260620 09:00:22.051326 7342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } }
I20260620 09:00:22.051471 7342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 060d38baef5d4da78af1530433048e73, State: Initialized, Role: LEARNER
I20260620 09:00:22.051585 7342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } }
I20260620 09:00:22.052095 7342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:22.052491 7345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:22.052713 7345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:00:22.055136 7359 catalog_manager.cc:1259] Loaded cluster ID: 4f93bd4c69cd41f1b8120d9e2c7af65f
I20260620 09:00:22.055253 7359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: loading cluster ID for follower catalog manager: success
I20260620 09:00:22.056540 6846 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
I20260620 09:00:22.057608 7359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: acquiring CA information for follower catalog manager: success
I20260620 09:00:22.059082 7359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 09:00:22.060024 7266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:22.060304 7266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:22.060408 7266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:22.064706 7266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:22.064795 7266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:22.064936 7266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:22.070288 7266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46691
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:22.071481 7266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:22.072647 7266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:22.082585 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.082578 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.082552 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.082870 7266 server_base.cc:1034] running on GCE node
I20260620 09:00:22.083359 7266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:22.083959 7266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:22.085120 7266 hybrid_clock.cc:648] HybridClock initialized: now 1781946022085082 us; error 50 us; skew 500 ppm
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:22.087908 7266 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:00:22.088821 7266 webserver.cc:460] Webserver started at http://127.0.98.1:46691/ using document root <none> and password file <none>
I20260620 09:00:22.089445 7266 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:22.089524 7266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:22.093394 7266 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 09:00:22.095655 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.097122 7266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:22.097267 7266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "1fa651c62fe848f39e269be2fdab4f57"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "0f523c7b1fecf42e6edd81143be28202"
server_key_iv: "b682a55470c3ca8c19194ef0c291c16b"
server_key_version: "encryptionkey@0"
I20260620 09:00:22.097810 7266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:22.115383 7266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:22.118516 7266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:22.118727 7266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:22.119369 7266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:22.120347 7266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:22.120393 7266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.120496 7266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:22.120528 7266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.130159 7266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40363
I20260620 09:00:22.130187 7484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40363 every 8 connection(s)
I20260620 09:00:22.131224 7266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:22.141141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 7266
I20260620 09:00:22.141366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6717
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:00:22.148404 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.133282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55983 (local address 127.0.98.62:36813)
0620 09:00:22.133398 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.133404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.134478 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 09:00:22.135743 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.135746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.135796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.135850 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.137818 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138367 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.139708 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.140037 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140121 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.143938 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.143942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.143945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.143977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.145737 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.146314 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.146318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.146320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.146409 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.146885 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.146891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.146892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.147142 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.147217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.148118 (+ 901us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.148240 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:00:22.148404 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.133190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54793 (local address 127.0.98.61:34855)
0620 09:00:22.133406 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.133411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.134810 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 09:00:22.135734 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.135738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.135796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.135850 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.137831 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138625 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.139634 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.139951 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140050 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.143976 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.143983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.143990 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.144023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.146379 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.146908 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.146911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.146913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.146987 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.147316 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.147319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.147321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.147513 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.147582 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.148118 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.148249 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 09:00:22.148630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36813
--webserver_port=38815
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:00:22.150837 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.133709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:55983)
0620 09:00:22.134318 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.134362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.135492 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.136157 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.136166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.136835 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.137666 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.137683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138506 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.138510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.139566 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.139575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.140243 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.140313 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.141096 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.141131 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.143757 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.145878 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.145884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.145901 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.146166 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.146541 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.146545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.146549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.146746 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.147312 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.147319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.147850 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.149178 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.149656 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":455,"spinlock_wait_cycles":37376,"thread_start_us":756,"threads_started":1}
I20260620 09:00:22.151119 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.133709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.98.1:54793)
0620 09:00:22.134721 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.134734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.135492 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.136157 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.136166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.136838 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.137677 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.137687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138754 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.138757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.139513 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.139522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.140093 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.140285 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.141050 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.141070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.143809 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.146495 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.146498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.146502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.146780 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.147100 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.147103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.147105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.147189 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.147653 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.147655 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.147832 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.148918 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.149703 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":262,"spinlock_wait_cycles":55680,"thread_start_us":183,"threads_started":1}
I20260620 09:00:22.152341 7486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:22.152376 7487 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34855
I20260620 09:00:22.152629 7487 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.152630 7486 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.153272 7487 heartbeater.cc:511] Master 127.0.98.61:34855 requested a full tablet report, sending...
I20260620 09:00:22.153270 7486 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
I20260620 09:00:22.154532 6520 ts_manager.cc:194] Re-registered known tserver with Master: 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363)
I20260620 09:00:22.155014 7122 ts_manager.cc:194] Registered new tserver with Master: 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363)
I20260620 09:00:22.155141 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55983
I20260620 09:00:22.156684 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.135038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.98.1:35301)
0620 09:00:22.135393 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.135407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.135508 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.136184 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.136195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.136835 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.137677 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.137687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138977 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.138982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.139785 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.139791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.140437 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.140576 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.145432 (+ 4856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.145453 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.148094 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.150287 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.150291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.150295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.150568 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.150900 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.150904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.150906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.150992 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.155992 (+ 5000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.155998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.156167 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.156292 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.156493 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":307,"spinlock_wait_cycles":53504,"thread_start_us":72,"threads_started":1}
I20260620 09:00:22.156891 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.135152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35301 (local address 127.0.98.60:43387)
0620 09:00:22.135470 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.135484 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.135498 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:22.135711 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.135714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.135766 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.135846 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.137836 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.138866 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.139895 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.140207 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.140300 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.148279 (+ 7979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.148283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.148286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.148315 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.150138 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.150698 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.150703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.150706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.150796 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.151087 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.151092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.151094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.151325 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.156038 (+ 4713us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.156392 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.156596 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260620 09:00:22.157462 7485 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43387
I20260620 09:00:22.157546 7485 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.157737 7485 heartbeater.cc:511] Master 127.0.98.60:43387 requested a full tablet report, sending...
I20260620 09:00:22.158707 7298 ts_manager.cc:194] Registered new tserver with Master: 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363)
W20260620 09:00:22.265952 7493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:22.266285 7493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:22.266404 7493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:22.270545 7493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:22.270666 7493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:22.270730 7493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:22.276298 7493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38815
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:22.277398 7493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:22.277729 6980 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
I20260620 09:00:22.278573 7493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:00:22.288748 7493 server_base.cc:1034] running on GCE node
W20260620 09:00:22.288764 7501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.289023 7499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.288733 7498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.289469 7493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:22.290108 7493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:22.291324 7493 hybrid_clock.cc:648] HybridClock initialized: now 1781946022291284 us; error 48 us; skew 500 ppm
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:22.294093 7493 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:00:22.295053 7493 webserver.cc:460] Webserver started at http://127.0.98.2:38815/ using document root <none> and password file <none>
I20260620 09:00:22.295583 7493 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:22.295652 7493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:22.299894 7493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:22.302435 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.303763 7493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:22.303894 7493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "4d762d9cd479416da10b933a0a94eacf"
format_stamp: "Formatted at 2026-06-20 09:00:20 on dist-test-slave-scsf"
server_key: "14aea3160df9ccdf39385548f89fa826"
server_key_iv: "9b9be922c76f8a7636bccbc853bbfcbd"
server_key_version: "encryptionkey@0"
I20260620 09:00:22.304553 7493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:22.318889 7493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:22.322593 7493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:22.322803 7493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:22.323444 7493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:22.324334 7493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:22.324383 7493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.324422 7493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:22.324502 7493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.334040 7493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36283
I20260620 09:00:22.334062 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36283 every 8 connection(s)
I20260620 09:00:22.335150 7493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:22.336217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 7493
I20260620 09:00:22.336406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6851
I20260620 09:00:22.347194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40699
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36813
--webserver_port=37079
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:00:22.353929 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.345799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55942)
0620 09:00:22.345918 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.345933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.346009 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.348318 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.348321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.348348 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.348489 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.348501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.350886 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.350891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.352279 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.352295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.353487 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.353491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.353591 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.353625 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.353679 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.353767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:00:22.354368 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.337501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56617 (local address 127.0.98.60:43387)
0620 09:00:22.337636 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.337640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.338714 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 09:00:22.339801 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.339804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.339859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.339938 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.342124 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.343007 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.345043 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.345506 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.345601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.349443 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.349449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.349454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.349483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.351462 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.352971 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.352974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.352976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.353066 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.353369 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.353375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.353380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.353584 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.353668 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.354049 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.354219 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:22.354666 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.346097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.98.60:43387)
0620 09:00:22.348028 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.348033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.348045 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:00:22.348095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.348099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.348158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.348609 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.348727 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.350736 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.352500 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.353338 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.354285 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.354438 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.354473 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.354549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":92,"threads_started":1}
I20260620 09:00:22.355194 7308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 09:00:22.355374 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.338008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.98.2:56617)
0620 09:00:22.338569 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.338613 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.339587 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.341444 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.341447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.341626 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.341942 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.341967 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.343231 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.343237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.344897 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.344912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.345815 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.345822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.346074 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.346787 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.346812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.349294 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.352512 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.352515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.352517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.352840 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.353168 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.353171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.353173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.353264 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.353745 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.353747 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.353832 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.354548 (+ 716us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.354709 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":149,"thread_start_us":151,"threads_started":1}
I20260620 09:00:22.355757 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.339172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:47761)
0620 09:00:22.339601 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.339615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.339724 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.340564 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.340574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.341195 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.341941 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.341965 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.343242 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.343245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.345266 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.345279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.346345 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.346349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.346465 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.347013 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.347033 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.349199 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.351605 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.351611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.351644 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.352041 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.352415 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.352420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.352425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.352656 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.353189 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.353198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.353573 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.353805 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.354174 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":89,"thread_start_us":85,"threads_started":1}
I20260620 09:00:22.355722 7345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23429c57255a498f99daa1ec36d040d8. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:22.355866 7345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:00:22.356408 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.339324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47761 (local address 127.0.98.62:36813)
0620 09:00:22.339451 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.339455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.339722 (+ 267us) server_negotiation.cc:407] Connection header received
0620 09:00:22.340149 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.340155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.340220 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.340317 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.342135 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.343057 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.345425 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.345847 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.345927 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.349364 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.349369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.349372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.349406 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.351472 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.352164 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.352169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.352176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.352291 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.352788 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.352793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.352797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.353039 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.353131 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.356124 (+ 2993us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.356265 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:00:22.357080 7623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:22.357249 7622 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43387
I20260620 09:00:22.357383 7623 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.357381 7622 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.358048 7622 heartbeater.cc:511] Master 127.0.98.60:43387 requested a full tablet report, sending...
I20260620 09:00:22.358049 7623 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
I20260620 09:00:22.359272 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.343894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.98.2:55645)
0620 09:00:22.346830 (+ 2936us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.346843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.347752 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.348218 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.348224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.348550 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.348709 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.348720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.349843 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.349847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.350797 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.350811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.351644 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.351647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.351755 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.352278 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.352296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.353926 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.357556 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.357559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.357561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.357813 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.358136 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.358138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.358140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.358273 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.358736 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.358738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.358821 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.358909 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.359054 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2858,"thread_start_us":117,"threads_started":1}
I20260620 09:00:22.359433 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.344276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55645 (local address 127.0.98.61:34855)
0620 09:00:22.344388 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.344392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.346921 (+ 2529us) server_negotiation.cc:407] Connection header received
0620 09:00:22.347912 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.347919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.347987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.348071 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.348851 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.349653 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.350940 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.351333 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.351424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.354274 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.354278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.354280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.354307 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.357341 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.357931 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.357935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.357937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.358023 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.358380 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.358383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.358384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.358590 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.358681 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.359034 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.359211 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:00:22.359485 7298 ts_manager.cc:194] Registered new tserver with Master: 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
I20260620 09:00:22.360000 6520 ts_manager.cc:194] Re-registered known tserver with Master: 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
I20260620 09:00:22.360198 7624 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34855
I20260620 09:00:22.360320 7624 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.360554 7624 heartbeater.cc:511] Master 127.0.98.61:34855 requested a full tablet report, sending...
I20260620 09:00:22.360780 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47761
I20260620 09:00:22.361382 7122 ts_manager.cc:194] Registered new tserver with Master: 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
W20260620 09:00:22.475492 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:22.475807 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:22.475893 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:22.480326 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:22.480422 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:22.480548 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:22.486433 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37079
--tserver_master_addrs=127.0.98.62:36813,127.0.98.61:34855,127.0.98.60:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:22.487576 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:22.488786 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:22.498875 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.498947 7630 server_base.cc:1034] running on GCE node
W20260620 09:00:22.499267 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:22.498878 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:22.499949 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:22.500818 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:22.502012 7630 hybrid_clock.cc:648] HybridClock initialized: now 1781946022501954 us; error 66 us; skew 500 ppm
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:22.505612 7630 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:00:22.506560 7630 webserver.cc:460] Webserver started at http://127.0.98.3:37079/ using document root <none> and password file <none>
I20260620 09:00:22.507153 7630 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:22.507296 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:22.512204 7630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:22.514840 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.516482 7630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:22.516597 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "3f05f49e630b472db248ebaa113915f1"
format_stamp: "Formatted at 2026-06-20 09:00:21 on dist-test-slave-scsf"
server_key: "84a9b3432996edff942a3872a2ecd566"
server_key_iv: "88c637ad73c59d3b35d873e3a64fd72b"
server_key_version: "encryptionkey@0"
I20260620 09:00:22.517292 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:22.535588 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:22.538852 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:22.539018 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:22.539558 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:22.540388 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:22.540438 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.540552 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:22.540575 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:22.550087 7630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40699
I20260620 09:00:22.550104 7758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40699 every 8 connection(s)
I20260620 09:00:22.551167 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:22.561208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 7630
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 09:00:22 dist-test-slave-scsf krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:00:22.567947 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.553222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50057 (local address 127.0.98.60:43387)
0620 09:00:22.553351 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.553355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.554736 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 09:00:22.555843 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.555847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.555894 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.555964 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.557484 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.558251 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.559602 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.559945 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.560009 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.563883 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.563890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.563895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.563923 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.565826 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566300 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.566303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.566305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.566384 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566637 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.566640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.566641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.566850 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.566931 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.567624 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.567745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:00:22.568885 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.553734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.98.3:48117)
0620 09:00:22.554365 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.554410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.555604 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.556149 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.556157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.556720 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.557326 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.557332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.558391 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.558395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.559467 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.559477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.560157 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.560162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.560321 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.560973 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.560990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.562970 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.565561 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.565567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.565584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.565975 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.566258 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.566264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.566431 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.566875 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.566882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.567212 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.567417 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.567774 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":209,"spinlock_wait_cycles":13184,"thread_start_us":672,"threads_started":1}
I20260620 09:00:22.569463 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.553734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.98.3:50057)
0620 09:00:22.554653 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.554666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.555604 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.556149 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.556157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.556720 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.557324 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.557332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.558391 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.558395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.559467 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.559477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.560176 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.560179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.560297 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.560910 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.560941 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.563464 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.565935 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.565937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.565939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.566178 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.566474 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.566478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.566554 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.566973 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.566976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.567235 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.567416 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.568210 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":790,"spinlock_wait_cycles":95616,"thread_start_us":181,"threads_started":1}
I20260620 09:00:22.570128 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.553191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48117 (local address 127.0.98.61:34855)
0620 09:00:22.553496 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.553500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.554524 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 09:00:22.555837 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.555841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.555894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.555964 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.557484 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.558259 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.559602 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.559936 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.560009 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.563156 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.563160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.563166 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.563200 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.565425 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566080 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.566083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.566085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.566165 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.566529 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.566532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.566534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.566722 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.566822 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.569865 (+ 3043us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.569980 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260620 09:00:22.570541 7759 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43387
I20260620 09:00:22.570779 7759 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.571399 7759 heartbeater.cc:511] Master 127.0.98.60:43387 requested a full tablet report, sending...
I20260620 09:00:22.571966 7761 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34855
I20260620 09:00:22.572048 7761 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.572213 7761 heartbeater.cc:511] Master 127.0.98.61:34855 requested a full tablet report, sending...
I20260620 09:00:22.573074 7122 ts_manager.cc:194] Registered new tserver with Master: 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:22.573873 7298 ts_manager.cc:194] Registered new tserver with Master: 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:22.574066 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.555118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53007 (local address 127.0.98.62:36813)
0620 09:00:22.555255 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.555260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.555379 (+ 119us) server_negotiation.cc:407] Connection header received
0620 09:00:22.555823 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.555827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.555880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.555974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.557469 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.557994 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.559155 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.559394 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.559521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.570111 (+ 10590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.570114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.570117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.570139 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.572227 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.572721 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.572724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.572727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.572811 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.573157 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.573160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.573162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.573380 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.573454 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.573783 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.573909 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:00:22.574308 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.563969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.98.61:34855)
0620 09:00:22.564200 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.564206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.564700 (+ 494us) server_negotiation.cc:407] Connection header received
0620 09:00:22.564748 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.564751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.564811 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.564900 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.565929 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.566681 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.567885 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.568216 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.568309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.570108 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.570112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.570115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.570138 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.572055 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.572562 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.572567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.572570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.572649 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.573003 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.573008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.573013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.573230 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.573345 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.574034 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.574158 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260620 09:00:22.574555 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.555011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:53007)
0620 09:00:22.555283 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.555303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.555611 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.556148 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.556155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.556710 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:22.557308 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.557322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.558133 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.558136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.559027 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.559035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.559562 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.559565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.559741 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.560340 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:22.560385 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:22.569963 (+ 9578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:22.572342 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.572345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.572347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.572610 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.572960 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:22.572965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:22.572970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:22.573054 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:22.573509 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:22.573513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:22.573605 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.574218 (+ 613us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.574360 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":38,"spinlock_wait_cycles":63616,"thread_start_us":103,"threads_started":1}
I20260620 09:00:22.575280 7760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36813
I20260620 09:00:22.575399 7760 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:22.575562 7760 heartbeater.cc:511] Master 127.0.98.62:36813 requested a full tablet report, sending...
I20260620 09:00:22.576287 6520 ts_manager.cc:194] Re-registered known tserver with Master: 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:22.577123 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53007
I20260620 09:00:22.581072 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.563794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.60:43387)
0620 09:00:22.563917 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.563922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.570806 (+ 6884us) server_negotiation.cc:407] Connection header received
0620 09:00:22.571022 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.571025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.571067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.571113 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.572014 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.573089 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.574909 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.575257 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.575362 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.577096 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.577100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.577103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.577128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.578748 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.579648 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.579651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.579652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.579727 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.580035 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.580043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.580044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.580214 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.580361 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.580733 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.580844 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:00:22.581208 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.563653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.0.98.62:36813)
0620 09:00:22.564660 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.564664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.564679 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:22.564720 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.564726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.564787 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.565815 (+ 1028us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.565926 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.566685 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.567901 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.568318 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.568754 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.575828 (+ 7074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.575832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.575834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.575865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.577529 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.579601 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.579608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.579610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.579705 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.580002 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.580007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.580009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.580191 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.580331 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.580670 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.580983 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":92,"threads_started":1}
I20260620 09:00:22.583828 6520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:22.586926 6520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:22.589454 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: true } } }
I20260620 09:00:22.589627 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:22.592937 7778 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: attempting to promote NON_VOTER 060d38baef5d4da78af1530433048e73 to VOTER
I20260620 09:00:22.593338 7778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:22.594539 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:00:22.594929 7270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:00:22.595763 7308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 LEARNER]: Refusing update from remote peer 23429c57255a498f99daa1ec36d040d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:00:22.596371 7778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:00:22.596531 7270 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 060d38baef5d4da78af1530433048e73 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.597687 7308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 060d38baef5d4da78af1530433048e73 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.598021 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.598112 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:22.598253 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.598356 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:22.598717 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 060d38baef5d4da78af1530433048e73 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.599606 7345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.599751 7345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:22.600204 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.600303 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:22.601477 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23429c57255a498f99daa1ec36d040d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:22.601840 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:22.608570 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.601771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40363 (local address 127.0.0.1:41140)
0620 09:00:22.601895 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.601911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.601995 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.604275 (+ 2280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.604279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.604294 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.604413 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.604420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.605914 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.605918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.607024 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.607033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.608130 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.608133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.608245 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.608272 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.608323 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.608385 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":33}
I20260620 09:00:22.609386 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.602809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.1:40363)
0620 09:00:22.603151 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.603168 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.603200 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:00:22.603286 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.603292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.603536 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.604566 (+ 1030us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.604718 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.605815 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.607137 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.608003 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.608874 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.609058 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.609119 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.609240 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260620 09:00:22.609719 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.602066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40699 (local address 127.0.0.1:48126)
0620 09:00:22.602653 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.602664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.602822 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.604147 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.604153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.604209 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.604414 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.604420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.607122 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.607124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.607851 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.607858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.609297 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.609301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.609391 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.609422 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.609504 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.609577 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":23,"thread_start_us":107,"threads_started":1}
I20260620 09:00:22.610150 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.602421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36283 (local address 127.0.0.1:54674)
0620 09:00:22.603061 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.603074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.603159 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.604854 (+ 1695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.604857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.604876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.605022 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.605031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.606834 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.606837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.607922 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.607933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.609696 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.609699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.609796 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.609830 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.609894 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.609959 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":85,"threads_started":1}
I20260620 09:00:22.610162 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.602225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.0.98.3:40699)
0620 09:00:22.603414 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.603426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.603449 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:00:22.603528 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.603533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.603765 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.604173 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.604594 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.605815 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.607979 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.609113 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.609258 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.609404 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.609663 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.610018 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":111,"threads_started":1}
I20260620 09:00:22.610695 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.602926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.0.98.2:36283)
0620 09:00:22.603531 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.603542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.603559 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:00:22.603619 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.603623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.604580 (+ 957us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.605178 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.605330 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.606702 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.608065 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.609542 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.610312 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.610476 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.610518 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.610587 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":105,"threads_started":1}
I20260620 09:00:22.612730 7419 tablet_service.cc:1467] Processing CreateTablet for tablet bc191a2136544207ba222c4f600aae50 (DEFAULT_TABLE table=test-table [id=2a86a3dcb7ac436fb0a8b0133002c092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:22.613336 7556 tablet_service.cc:1467] Processing CreateTablet for tablet bc191a2136544207ba222c4f600aae50 (DEFAULT_TABLE table=test-table [id=2a86a3dcb7ac436fb0a8b0133002c092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:22.613909 7419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc191a2136544207ba222c4f600aae50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:22.614367 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc191a2136544207ba222c4f600aae50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:22.618835 7693 tablet_service.cc:1467] Processing CreateTablet for tablet bc191a2136544207ba222c4f600aae50 (DEFAULT_TABLE table=test-table [id=2a86a3dcb7ac436fb0a8b0133002c092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:22.619737 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc191a2136544207ba222c4f600aae50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:22.622349 7792 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Bootstrap starting.
I20260620 09:00:22.624085 7793 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Bootstrap starting.
I20260620 09:00:22.625754 7792 tablet_bootstrap.cc:654] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:22.625887 7793 tablet_bootstrap.cc:654] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:22.626551 7793 log.cc:824] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:22.626603 7792 log.cc:824] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:22.627848 7796 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Bootstrap starting.
I20260620 09:00:22.628639 7793 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: No bootstrap required, opened a new log
I20260620 09:00:22.628639 7792 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: No bootstrap required, opened a new log
I20260620 09:00:22.628821 7792 ts_tablet_manager.cc:1397] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 09:00:22.628821 7793 ts_tablet_manager.cc:1397] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:00:22.629700 7796 tablet_bootstrap.cc:654] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:22.630483 7796 log.cc:824] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:22.632345 7796 tablet_bootstrap.cc:492] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: No bootstrap required, opened a new log
I20260620 09:00:22.632597 7796 ts_tablet_manager.cc:1397] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:00:22.634079 7793 raft_consensus.cc:348] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.634079 7792 raft_consensus.cc:348] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.634369 7793 raft_consensus.cc:374] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:22.634369 7792 raft_consensus.cc:374] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:22.634430 7792 raft_consensus.cc:729] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d762d9cd479416da10b933a0a94eacf, State: Initialized, Role: FOLLOWER
I20260620 09:00:22.634438 7793 raft_consensus.cc:729] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa651c62fe848f39e269be2fdab4f57, State: Initialized, Role: FOLLOWER
I20260620 09:00:22.634963 7792 consensus_queue.cc:260] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.634963 7793 consensus_queue.cc:260] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.636042 7792 ts_tablet_manager.cc:1428] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 09:00:22.636088 7793 ts_tablet_manager.cc:1428] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 09:00:22.636312 7486 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
I20260620 09:00:22.637115 7796 raft_consensus.cc:348] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.637385 7796 raft_consensus.cc:374] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:22.637434 7796 raft_consensus.cc:729] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f05f49e630b472db248ebaa113915f1, State: Initialized, Role: FOLLOWER
I20260620 09:00:22.637894 7796 consensus_queue.cc:260] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.638137 7623 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
I20260620 09:00:22.639438 7796 ts_tablet_manager.cc:1428] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:00:22.640093 7760 heartbeater.cc:503] Master 127.0.98.62:36813 was elected leader, sending a full tablet report...
I20260620 09:00:22.725359 7800 raft_consensus.cc:482] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:22.725553 7800 raft_consensus.cc:504] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.726605 7800 leader_election.cc:288] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363), 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
I20260620 09:00:22.730363 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.726979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59873 (local address 127.0.98.1:40363)
0620 09:00:22.727129 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.727134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.727218 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:00:22.727280 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.727283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.727340 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.727417 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.727696 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.728202 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729397 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.729796 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729913 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.729978 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.730159 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.730226 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:00:22.730386 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.726856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40363 (local address 127.0.98.3:59873)
0620 09:00:22.726991 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.727009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.727108 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.727432 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.727436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.727454 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.727574 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.727580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.728318 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.728325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.729269 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.729278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.729914 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.730018 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.730040 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.730138 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.730210 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":48}
I20260620 09:00:22.730415 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.727160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36283 (local address 127.0.98.3:42713)
0620 09:00:22.727254 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:22.727268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:22.727362 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:22.727660 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:22.727663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:22.727677 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:22.727789 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.727794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.728571 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.728574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.729339 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:22.729347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.729984 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:22.730087 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.730112 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:00:22.730198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:00:22.730266 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":15}
I20260620 09:00:22.730415 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.727302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42713 (local address 127.0.98.2:36283)
0620 09:00:22.727457 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.727462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.727475 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:22.727526 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.727528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.727566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.727649 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:22.727905 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.728403 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729468 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.729866 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.729981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.730042 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.730225 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.730308 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:00:22.731025 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc191a2136544207ba222c4f600aae50" candidate_uuid: "3f05f49e630b472db248ebaa113915f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa651c62fe848f39e269be2fdab4f57" is_pre_election: true
I20260620 09:00:22.731173 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc191a2136544207ba222c4f600aae50" candidate_uuid: "3f05f49e630b472db248ebaa113915f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d762d9cd479416da10b933a0a94eacf" is_pre_election: true
I20260620 09:00:22.731370 7439 raft_consensus.cc:2454] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f05f49e630b472db248ebaa113915f1 in term 0.
I20260620 09:00:22.731503 7576 raft_consensus.cc:2454] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f05f49e630b472db248ebaa113915f1 in term 0.
I20260620 09:00:22.731899 7648 leader_election.cc:302] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa651c62fe848f39e269be2fdab4f57, 3f05f49e630b472db248ebaa113915f1; no voters:
I20260620 09:00:22.732203 7800 raft_consensus.cc:2781] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:22.732285 7800 raft_consensus.cc:482] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:22.732316 7800 raft_consensus.cc:3037] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:22.733404 7800 raft_consensus.cc:504] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.733765 7800 leader_election.cc:288] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363), 4d762d9cd479416da10b933a0a94eacf (127.0.98.2:36283)
I20260620 09:00:22.734267 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc191a2136544207ba222c4f600aae50" candidate_uuid: "3f05f49e630b472db248ebaa113915f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d762d9cd479416da10b933a0a94eacf"
I20260620 09:00:22.734269 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc191a2136544207ba222c4f600aae50" candidate_uuid: "3f05f49e630b472db248ebaa113915f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa651c62fe848f39e269be2fdab4f57"
I20260620 09:00:22.734378 7576 raft_consensus.cc:3037] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:22.734406 7439 raft_consensus.cc:3037] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:22.735499 7576 raft_consensus.cc:2454] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f05f49e630b472db248ebaa113915f1 in term 1.
I20260620 09:00:22.735505 7439 raft_consensus.cc:2454] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f05f49e630b472db248ebaa113915f1 in term 1.
I20260620 09:00:22.735818 7648 leader_election.cc:302] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa651c62fe848f39e269be2fdab4f57, 3f05f49e630b472db248ebaa113915f1; no voters:
I20260620 09:00:22.736045 7800 raft_consensus.cc:2781] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:22.736346 7800 raft_consensus.cc:686] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3f05f49e630b472db248ebaa113915f1, State: Running, Role: LEADER
I20260620 09:00:22.736739 7800 consensus_queue.cc:237] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:22.740338 6520 catalog_manager.cc:5495] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f05f49e630b472db248ebaa113915f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3f05f49e630b472db248ebaa113915f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:22.760286 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.757114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.98.3:40699)
0620 09:00:22.757259 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.757262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.757275 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:22.757349 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.757351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.757404 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.757491 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:22.757689 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.758160 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.759040 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.759246 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.759321 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.759555 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:22.759992 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:22.760073 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.760133 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.760178 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:00:22.766095 7576 raft_consensus.cc:1261] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Refusing update from remote peer 3f05f49e630b472db248ebaa113915f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:22.766095 7439 raft_consensus.cc:1261] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Refusing update from remote peer 3f05f49e630b472db248ebaa113915f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:22.766862 7801 consensus_queue.cc:1035] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:22.767042 7800 consensus_queue.cc:1035] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:22.775643 7805 mvcc.cc:205] Tried to move back new op lower bound from 7298850909242671104 to 7298850909132644352. Current Snapshot: MvccSnapshot[applied={T|T < 7298850909242671104}]
I20260620 09:00:22.786440 6520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:22.786635 6520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:22.791904 6520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23429c57255a498f99daa1ec36d040d8: Sending DeleteTablet for 3 replicas of tablet bc191a2136544207ba222c4f600aae50
I20260620 09:00:22.792754 7419 tablet_service.cc:1514] Processing DeleteTablet for tablet bc191a2136544207ba222c4f600aae50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41140
I20260620 09:00:22.792784 7556 tablet_service.cc:1514] Processing DeleteTablet for tablet bc191a2136544207ba222c4f600aae50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54674
I20260620 09:00:22.792773 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet bc191a2136544207ba222c4f600aae50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48126
I20260620 09:00:22.793325 7814 tablet_replica.cc:330] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: stopping tablet replica
I20260620 09:00:22.793396 7816 tablet_replica.cc:330] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: stopping tablet replica
I20260620 09:00:22.793632 7815 tablet_replica.cc:330] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: stopping tablet replica
I20260620 09:00:22.793767 7816 raft_consensus.cc:2229] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:22.794113 7816 raft_consensus.cc:2258] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:22.794396 7815 raft_consensus.cc:2229] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:22.794126 7814 raft_consensus.cc:2229] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:22.794946 7815 raft_consensus.cc:2258] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:22.795023 7814 raft_consensus.cc:2258] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:22.796061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6489
I20260620 09:00:22.803328 7815 ts_tablet_manager.cc:1905] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:22.803838 7814 ts_tablet_manager.cc:1905] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:22.804797 7816 ts_tablet_manager.cc:1905] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:22.804971 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.796038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.98.61:34855)
0620 09:00:22.796148 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.796154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.796168 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:22.796212 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.796217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.796274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.796352 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.797586 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.798102 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.798962 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.799207 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.799323 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.800905 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.800911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.800917 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.800949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.802782 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.803276 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.803280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.803281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.803367 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.803897 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.803904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.803909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.804177 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.804277 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.804681 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.804809 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:00:22.807422 7816 ts_tablet_manager.cc:1918] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:22.807530 7816 log.cc:1196] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/bc191a2136544207ba222c4f600aae50
I20260620 09:00:22.807864 7816 ts_tablet_manager.cc:1939] T bc191a2136544207ba222c4f600aae50 P 1fa651c62fe848f39e269be2fdab4f57: Deleting consensus metadata
I20260620 09:00:22.809409 7815 ts_tablet_manager.cc:1918] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:22.810506 7814 ts_tablet_manager.cc:1918] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:22.810679 7814 log.cc:1196] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/bc191a2136544207ba222c4f600aae50
I20260620 09:00:22.811120 7814 ts_tablet_manager.cc:1939] T bc191a2136544207ba222c4f600aae50 P 4d762d9cd479416da10b933a0a94eacf: Deleting consensus metadata
I20260620 09:00:22.811268 7815 log.cc:1196] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/bc191a2136544207ba222c4f600aae50
I20260620 09:00:22.811676 7815 ts_tablet_manager.cc:1939] T bc191a2136544207ba222c4f600aae50 P 3f05f49e630b472db248ebaa113915f1: Deleting consensus metadata
W20260620 09:00:22.813076 7510 connection.cc:441] server connection from 127.0.0.1:54674 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54674 (request call id 1) could send its response
W20260620 09:00:22.813148 7373 connection.cc:441] server connection from 127.0.0.1:41140 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41140 (request call id 1) could send its response
W20260620 09:00:22.814257 7647 connection.cc:441] server connection from 127.0.0.1:48126 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48126 (request call id 1) could send its response
I20260620 09:00:22.816548 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.795887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.0.98.60:43387)
0620 09:00:22.796019 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.796023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.806384 (+ 10361us) server_negotiation.cc:407] Connection header received
0620 09:00:22.806579 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.806584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.806644 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.806713 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.807596 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.808367 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.809836 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.810180 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.810263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.812144 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.812147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.812150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.812175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.814252 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.815013 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.815017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.815018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.815099 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.815414 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.815420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.815421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.815651 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.815777 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.816096 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.816238 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:00:22.823951 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.814864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.98.60:43387)
0620 09:00:22.814988 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.814994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.815586 (+ 592us) server_negotiation.cc:407] Connection header received
0620 09:00:22.815751 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.815754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.815807 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.815850 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.816754 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.817390 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.818210 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.818439 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.818493 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.820209 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.820213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.820215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.820235 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.822011 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.822470 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.822476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.822480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.822585 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.822977 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.822981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.822983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.823217 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.823328 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.823663 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.823773 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:00:22.825284 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:22.814497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.0.98.61:34855)
0620 09:00:22.814622 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:22.814626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:22.815276 (+ 650us) server_negotiation.cc:407] Connection header received
0620 09:00:22.815449 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:22.815452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:22.815498 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:22.815545 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:22.817079 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.817842 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.819566 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:22.819953 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:22.820062 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:22.821600 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:22.821605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:22.821607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:22.821636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:22.823297 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.823760 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.823764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.823766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.823851 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:22.824141 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:22.824144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:22.824146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:22.824345 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:22.824440 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:22.824979 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 09:00:22.825075 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:00:23.779871 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:23.779343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:42837)
0620 09:00:23.779632 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:23.779754 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260620 09:00:23.779947 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:23.779350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:56109)
0620 09:00:23.779757 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:23.779844 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":71,"threads_started":1}
W20260620 09:00:23.780159 7760 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:23.780234 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:23.779492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:33493)
0620 09:00:23.780051 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:23.780129 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":54,"threads_started":1}
W20260620 09:00:23.780324 7486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
W20260620 09:00:23.780443 7623 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:24.291229 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23429c57255a498f99daa1ec36d040d8)
I20260620 09:00:24.291514 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:24.291613 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23429c57255a498f99daa1ec36d040d8)
I20260620 09:00:24.291797 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:24.292781 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 060d38baef5d4da78af1530433048e73 (127.0.98.60:43387)
I20260620 09:00:24.292789 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 450e1074deee4223aab2b99b333566b8 (127.0.98.61:34855)
I20260620 09:00:24.293368 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.292901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47622)
0620 09:00:24.293204 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.293272 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":69,"threads_started":1}
I20260620 09:00:24.293541 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.292902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47620)
0620 09:00:24.293156 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.293462 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20260620 09:00:24.293524 7290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
W20260620 09:00:24.293690 7106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:24.294456 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.294204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47632)
0620 09:00:24.294341 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.294389 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 09:00:24.294593 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.294345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47648)
0620 09:00:24.294471 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.294517 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 09:00:24.294596 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
W20260620 09:00:24.294739 7106 leader_election.cc:334] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:24.297416 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.293380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.0.98.61:34855)
0620 09:00:24.293758 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:24.293762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:24.294107 (+ 345us) server_negotiation.cc:407] Connection header received
0620 09:00:24.294153 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:24.294155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:24.294194 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:24.294264 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:24.294811 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.295384 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296398 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.296860 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296978 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:24.297067 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:24.297188 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:00:24.297277 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":311,"thread_start_us":46,"threads_started":1}
I20260620 09:00:24.297430 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.293225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43387 (local address 127.0.0.1:55970)
0620 09:00:24.293933 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.293945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:24.294030 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:24.294464 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:24.294467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:24.294483 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:24.294566 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:24.294572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.295425 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.295430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:24.296220 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:24.296229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.296927 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:24.297070 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:24.297092 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:00:24.297184 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:00:24.297287 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":48,"thread_start_us":47,"threads_started":1}
I20260620 09:00:24.297509 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.293529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.60:43387)
0620 09:00:24.293715 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:24.293720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:24.294140 (+ 420us) server_negotiation.cc:407] Connection header received
0620 09:00:24.294198 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:24.294203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:24.294261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:24.294350 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:24.294694 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.295308 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296343 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.296809 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:24.296986 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:24.297199 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:00:24.297335 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260620 09:00:24.297842 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.293127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34855 (local address 127.0.0.1:46990)
0620 09:00:24.293808 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.293820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:24.293922 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:24.294384 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:24.294387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:24.294411 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:24.294595 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:24.294606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.295496 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.295503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:24.296277 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:24.296285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:24.296977 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:24.296981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:24.297067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:24.297096 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:00:24.297173 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:00:24.297323 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1}
I20260620 09:00:24.297930 7310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "450e1074deee4223aab2b99b333566b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "060d38baef5d4da78af1530433048e73" is_pre_election: true
I20260620 09:00:24.298151 7310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450e1074deee4223aab2b99b333566b8 in term 1.
I20260620 09:00:24.298292 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060d38baef5d4da78af1530433048e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "450e1074deee4223aab2b99b333566b8" is_pre_election: true
I20260620 09:00:24.298475 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060d38baef5d4da78af1530433048e73 in term 1.
I20260620 09:00:24.298496 7106 leader_election.cc:302] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 060d38baef5d4da78af1530433048e73, 450e1074deee4223aab2b99b333566b8; no voters: 23429c57255a498f99daa1ec36d040d8
I20260620 09:00:24.298700 7828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:00:24.298753 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 23429c57255a498f99daa1ec36d040d8)
I20260620 09:00:24.298790 7828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:00:24.298846 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 060d38baef5d4da78af1530433048e73, 450e1074deee4223aab2b99b333566b8; no voters: 23429c57255a498f99daa1ec36d040d8
I20260620 09:00:24.299050 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:00:24.299117 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 23429c57255a498f99daa1ec36d040d8)
I20260620 09:00:24.299165 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:00:24.300125 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:24.300334 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:24.300585 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 election: Requested vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 060d38baef5d4da78af1530433048e73 (127.0.98.60:43387)
I20260620 09:00:24.300707 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 election: Requested vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 450e1074deee4223aab2b99b333566b8 (127.0.98.61:34855)
I20260620 09:00:24.301065 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060d38baef5d4da78af1530433048e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "450e1074deee4223aab2b99b333566b8"
I20260620 09:00:24.301087 7310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "450e1074deee4223aab2b99b333566b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "060d38baef5d4da78af1530433048e73"
I20260620 09:00:24.301229 7131 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 060d38baef5d4da78af1530433048e73 in current term 2: Already voted for candidate 450e1074deee4223aab2b99b333566b8 in this term.
I20260620 09:00:24.301229 7310 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 450e1074deee4223aab2b99b333566b8 in current term 2: Already voted for candidate 060d38baef5d4da78af1530433048e73 in this term.
I20260620 09:00:24.301322 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.300737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47654)
0620 09:00:24.301217 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.301263 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260620 09:00:24.301309 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.301066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47666)
0620 09:00:24.301160 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.301212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 09:00:24.302143 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.301922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47676)
0620 09:00:24.302012 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.302056 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 09:00:24.302254 7106 leader_election.cc:334] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:24.302349 7106 leader_election.cc:302] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 450e1074deee4223aab2b99b333566b8; no voters: 060d38baef5d4da78af1530433048e73, 23429c57255a498f99daa1ec36d040d8
I20260620 09:00:24.302497 7828 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:00:24.302644 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.301953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47690)
0620 09:00:24.302063 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.302203 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:00:24.302747 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:24.302846 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 060d38baef5d4da78af1530433048e73; no voters: 23429c57255a498f99daa1ec36d040d8, 450e1074deee4223aab2b99b333566b8
I20260620 09:00:24.302997 7829 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:00:24.781263 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.780742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:52705)
0620 09:00:24.781065 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.781143 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260620 09:00:24.781653 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.780851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:50975)
0620 09:00:24.781258 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.781323 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":68,"threads_started":1}
I20260620 09:00:24.781653 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:24.781134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:34171)
0620 09:00:24.781501 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:24.781556 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":72,"threads_started":1}
I20260620 09:00:25.782593 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.782084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:56225)
0620 09:00:25.782393 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.782463 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260620 09:00:25.782728 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.782296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:44903)
0620 09:00:25.782570 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.782631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260620 09:00:25.782847 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.782343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:50933)
0620 09:00:25.782700 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.782758 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":58,"threads_started":1}
I20260620 09:00:25.971684 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:25.971840 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:25.972235 7844 leader_election.cc:288] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 450e1074deee4223aab2b99b333566b8 (127.0.98.61:34855)
I20260620 09:00:25.972759 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060d38baef5d4da78af1530433048e73" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "450e1074deee4223aab2b99b333566b8" is_pre_election: true
I20260620 09:00:25.972961 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060d38baef5d4da78af1530433048e73 in term 2.
I20260620 09:00:25.972986 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.972543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47926)
0620 09:00:25.972767 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.972826 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260620 09:00:25.973320 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d38baef5d4da78af1530433048e73, 450e1074deee4223aab2b99b333566b8; no voters:
I20260620 09:00:25.973526 7844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 09:00:25.973587 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:25.973629 7844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:00:25.973860 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.973609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47932)
0620 09:00:25.973739 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.973798 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 09:00:25.973994 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:25.974741 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:25.975092 7844 leader_election.cc:288] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 election: Requested vote from peers 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813), 450e1074deee4223aab2b99b333566b8 (127.0.98.61:34855)
I20260620 09:00:25.975346 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.975158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47934)
0620 09:00:25.975247 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.975298 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:00:25.975498 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060d38baef5d4da78af1530433048e73" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "450e1074deee4223aab2b99b333566b8"
I20260620 09:00:25.975626 7131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:00:25.975993 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:25.975793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47936)
0620 09:00:25.975898 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:25.975947 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 09:00:25.976089 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
I20260620 09:00:25.976775 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060d38baef5d4da78af1530433048e73 in term 3.
I20260620 09:00:25.977104 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 060d38baef5d4da78af1530433048e73, 450e1074deee4223aab2b99b333566b8; no voters: 23429c57255a498f99daa1ec36d040d8
I20260620 09:00:25.977370 7844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:00:25.977579 7844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [term 3 LEADER]: Becoming Leader. State: Replica: 060d38baef5d4da78af1530433048e73, State: Running, Role: LEADER
I20260620 09:00:25.977842 7844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } }
I20260620 09:00:25.979249 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060d38baef5d4da78af1530433048e73. Latest consensus state: current_term: 3 leader_uuid: "060d38baef5d4da78af1530433048e73" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:25.979374 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:25.979676 7849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:25.981978 7849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:25.982429 7849 catalog_manager.cc:1259] Loaded cluster ID: 4f93bd4c69cd41f1b8120d9e2c7af65f
I20260620 09:00:25.982470 7849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:25.983064 7849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:25.983453 7849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Loaded TSK: 0
I20260620 09:00:25.984098 7849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:26.039850 7298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:26.043869 7298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:26.047041 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.046746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47966)
0620 09:00:26.046895 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.046961 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 09:00:26.047102 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [term 3 FOLLOWER]: Refusing update from remote peer 060d38baef5d4da78af1530433048e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 09:00:26.047588 7847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 09:00:26.047806 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.047556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.0.1:47970)
0620 09:00:26.047685 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.047735 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:00:26.048002 7290 consensus_peers.cc:487] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 -> Peer 23429c57255a498f99daa1ec36d040d8 (127.0.98.62:36813): Couldn't send request to peer 23429c57255a498f99daa1ec36d040d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:00:26.049232 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060d38baef5d4da78af1530433048e73. Latest consensus state: current_term: 3 leader_uuid: "060d38baef5d4da78af1530433048e73" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:26.049358 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:26.050522 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "060d38baef5d4da78af1530433048e73" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:26.050665 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:26.050900 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "060d38baef5d4da78af1530433048e73" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "23429c57255a498f99daa1ec36d040d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36813 } } peers { permanent_uuid: "450e1074deee4223aab2b99b333566b8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34855 } attrs { promote: false } } peers { permanent_uuid: "060d38baef5d4da78af1530433048e73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43387 } attrs { promote: false } } }
I20260620 09:00:26.051116 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450e1074deee4223aab2b99b333566b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:00:26.061619 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.057506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.0.98.3:40699)
0620 09:00:26.057757 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.057761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.057773 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:00:26.057820 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.057822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.057869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.058203 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:26.058306 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.059040 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.060349 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061028 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061204 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.061394 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.061463 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260620 09:00:26.061617 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.056763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40699 (local address 127.0.0.1:48142)
0620 09:00:26.056968 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.056984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:26.057063 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:26.057981 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:26.057984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:26.057997 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:26.058104 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.058110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.059140 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.059155 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.060218 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.060227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061155 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.061261 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061289 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:00:26.061365 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:00:26.061466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260620 09:00:26.061738 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.056568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.0.98.1:40363)
0620 09:00:26.057496 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.057500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.057513 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:26.057566 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.057568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.057624 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.057850 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:26.058090 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.058885 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.060433 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061076 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061439 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061497 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.061532 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.061599 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":93,"threads_started":1}
I20260620 09:00:26.062099 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.056433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40363 (local address 127.0.0.1:41142)
0620 09:00:26.056606 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.056621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:26.056726 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:26.057745 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:26.057748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:26.057765 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:26.057978 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.057983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.059004 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.059007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.060301 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.060311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061215 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.061291 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061309 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:00:26.061356 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:00:26.061959 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":649}
I20260620 09:00:26.062252 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.057101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36283 (local address 127.0.0.1:54678)
0620 09:00:26.057313 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.057326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:26.057409 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:26.057800 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:26.057802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:26.057816 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:26.057975 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.057981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.059701 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.059706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.060785 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.060793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061842 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.061936 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061957 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:00:26.062027 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:00:26.062105 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260620 09:00:26.062327 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.057203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.0.98.2:36283)
0620 09:00:26.057451 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.057456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.057472 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:26.057530 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.057533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.057585 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.057677 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:26.059044 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.059593 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.060942 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.061576 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.061690 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.061772 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.062070 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.062169 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260620 09:00:26.062608 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 54cd3a00030442798480ee6224e920b2 (DEFAULT_TABLE table=test-table [id=e48be2095f9140c7803b7f8be3718283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:26.062870 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cd3a00030442798480ee6224e920b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:26.063009 7556 tablet_service.cc:1467] Processing CreateTablet for tablet 54cd3a00030442798480ee6224e920b2 (DEFAULT_TABLE table=test-table [id=e48be2095f9140c7803b7f8be3718283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:26.063283 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cd3a00030442798480ee6224e920b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:26.064726 7419 tablet_service.cc:1467] Processing CreateTablet for tablet 54cd3a00030442798480ee6224e920b2 (DEFAULT_TABLE table=test-table [id=e48be2095f9140c7803b7f8be3718283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:26.064992 7419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cd3a00030442798480ee6224e920b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:26.065608 7862 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Bootstrap starting.
I20260620 09:00:26.066138 7863 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Bootstrap starting.
I20260620 09:00:26.066550 7862 tablet_bootstrap.cc:654] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:26.067122 7863 tablet_bootstrap.cc:654] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:26.067329 7865 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: Bootstrap starting.
I20260620 09:00:26.068275 7865 tablet_bootstrap.cc:654] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:26.068598 7862 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: No bootstrap required, opened a new log
I20260620 09:00:26.068614 7863 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: No bootstrap required, opened a new log
I20260620 09:00:26.068692 7863 ts_tablet_manager.cc:1397] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:26.068692 7862 ts_tablet_manager.cc:1397] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:26.069125 7862 raft_consensus.cc:348] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.069255 7862 raft_consensus.cc:374] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:26.069196 7863 raft_consensus.cc:348] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.069288 7862 raft_consensus.cc:729] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f05f49e630b472db248ebaa113915f1, State: Initialized, Role: FOLLOWER
I20260620 09:00:26.069295 7863 raft_consensus.cc:374] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:26.069331 7863 raft_consensus.cc:729] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d762d9cd479416da10b933a0a94eacf, State: Initialized, Role: FOLLOWER
I20260620 09:00:26.069432 7865 tablet_bootstrap.cc:492] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: No bootstrap required, opened a new log
I20260620 09:00:26.069435 7862 consensus_queue.cc:260] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.069446 7863 consensus_queue.cc:260] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.069514 7865 ts_tablet_manager.cc:1397] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:26.069787 7862 ts_tablet_manager.cc:1428] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:26.069787 7863 ts_tablet_manager.cc:1428] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:26.069950 7865 raft_consensus.cc:348] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.070073 7865 raft_consensus.cc:374] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:26.070112 7865 raft_consensus.cc:729] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa651c62fe848f39e269be2fdab4f57, State: Initialized, Role: FOLLOWER
I20260620 09:00:26.070428 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.070189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:46643)
0620 09:00:26.070292 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.070352 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:00:26.071043 7865 consensus_queue.cc:260] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.071146 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.070891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:46929)
0620 09:00:26.071037 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.071085 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 09:00:26.071453 7865 ts_tablet_manager.cc:1428] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:26.073143 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.072818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:37395)
0620 09:00:26.073011 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.073069 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 09:00:26.073411 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.073160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:53171)
0620 09:00:26.073275 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.073332 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 09:00:26.074234 7485 heartbeater.cc:503] Master 127.0.98.60:43387 was elected leader, sending a full tablet report...
I20260620 09:00:26.074563 7622 heartbeater.cc:503] Master 127.0.98.60:43387 was elected leader, sending a full tablet report...
I20260620 09:00:26.074767 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.074390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:54513)
0620 09:00:26.074624 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.074676 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260620 09:00:26.089386 7625 tablet.cc:2369] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:26.135694 7488 tablet.cc:2369] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:26.304829 7762 tablet.cc:2369] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:26.330192 7868 raft_consensus.cc:482] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:26.330353 7868 raft_consensus.cc:504] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.331504 7868 leader_election.cc:288] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363), 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:26.335601 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.331936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40363 (local address 127.0.98.2:42295)
0620 09:00:26.332076 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.332091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:26.332198 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:26.332528 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:26.332531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:26.332551 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:26.332665 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.332671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.333402 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.333406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.334290 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.334300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.335169 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.335173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.335273 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.335300 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:00:26.335385 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:00:26.335457 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":21}
I20260620 09:00:26.335623 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.332029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42295 (local address 127.0.98.1:40363)
0620 09:00:26.332158 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.332161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.332198 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:00:26.332303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.332306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.332360 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.332515 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:26.332782 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.333288 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.334436 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.335029 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.335135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.335197 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.335413 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.335506 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:00:26.336141 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cd3a00030442798480ee6224e920b2" candidate_uuid: "4d762d9cd479416da10b933a0a94eacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa651c62fe848f39e269be2fdab4f57" is_pre_election: true
I20260620 09:00:26.336300 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.332217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40699 (local address 127.0.98.2:50353)
0620 09:00:26.332624 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.332634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:26.332747 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:26.332984 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:26.332987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:26.333001 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:26.333113 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.333126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.333815 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.333821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.334699 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:26.334708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.335862 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.335866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:26.336019 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.336042 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:00:26.336115 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:00:26.336175 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260620 09:00:26.336292 7439 raft_consensus.cc:2454] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d762d9cd479416da10b933a0a94eacf in term 0.
I20260620 09:00:26.336326 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.332341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50353 (local address 127.0.98.3:40699)
0620 09:00:26.332528 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.332533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.332713 (+ 180us) server_negotiation.cc:407] Connection header received
0620 09:00:26.332833 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.332837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.332886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.332976 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:26.333228 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.333709 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.334820 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.335306 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.335420 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.335476 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.336135 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.336214 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 09:00:26.336721 7511 leader_election.cc:302] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa651c62fe848f39e269be2fdab4f57, 4d762d9cd479416da10b933a0a94eacf; no voters:
I20260620 09:00:26.337014 7868 raft_consensus.cc:2781] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:26.337110 7868 raft_consensus.cc:482] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:26.337169 7868 raft_consensus.cc:3037] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:26.337366 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cd3a00030442798480ee6224e920b2" candidate_uuid: "4d762d9cd479416da10b933a0a94eacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f05f49e630b472db248ebaa113915f1" is_pre_election: true
I20260620 09:00:26.337627 7713 raft_consensus.cc:2454] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d762d9cd479416da10b933a0a94eacf in term 0.
I20260620 09:00:26.338441 7868 raft_consensus.cc:504] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.338819 7868 leader_election.cc:288] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [CANDIDATE]: Term 1 election: Requested vote from peers 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363), 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699)
I20260620 09:00:26.339174 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cd3a00030442798480ee6224e920b2" candidate_uuid: "4d762d9cd479416da10b933a0a94eacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa651c62fe848f39e269be2fdab4f57"
I20260620 09:00:26.339263 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cd3a00030442798480ee6224e920b2" candidate_uuid: "4d762d9cd479416da10b933a0a94eacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f05f49e630b472db248ebaa113915f1"
I20260620 09:00:26.339322 7439 raft_consensus.cc:3037] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:26.339370 7713 raft_consensus.cc:3037] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:26.340265 7439 raft_consensus.cc:2454] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d762d9cd479416da10b933a0a94eacf in term 1.
I20260620 09:00:26.340472 7713 raft_consensus.cc:2454] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d762d9cd479416da10b933a0a94eacf in term 1.
I20260620 09:00:26.340545 7511 leader_election.cc:302] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa651c62fe848f39e269be2fdab4f57, 4d762d9cd479416da10b933a0a94eacf; no voters:
I20260620 09:00:26.340829 7868 raft_consensus.cc:2781] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:26.341092 7868 raft_consensus.cc:686] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 1 LEADER]: Becoming Leader. State: Replica: 4d762d9cd479416da10b933a0a94eacf, State: Running, Role: LEADER
I20260620 09:00:26.341526 7868 consensus_queue.cc:237] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } }
I20260620 09:00:26.341734 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.341519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:41243)
0620 09:00:26.341614 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.341672 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:00:26.343986 7305 catalog_manager.cc:5495] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d762d9cd479416da10b933a0a94eacf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d762d9cd479416da10b933a0a94eacf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d762d9cd479416da10b933a0a94eacf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36283 } health_report { overall_health: HEALTHY } } }
I20260620 09:00:26.356014 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.352645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.98.2:36283)
0620 09:00:26.352748 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:26.352752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:26.352773 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:00:26.352864 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:26.352866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:26.352920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:26.353013 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:26.353213 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.353890 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.354671 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:26.354870 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:26.354950 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:26.355251 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:26.355686 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:26.355772 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:26.355848 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:00:26.355894 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:00:26.358846 7439 raft_consensus.cc:1261] T 54cd3a00030442798480ee6224e920b2 P 1fa651c62fe848f39e269be2fdab4f57 [term 1 FOLLOWER]: Refusing update from remote peer 4d762d9cd479416da10b933a0a94eacf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:26.359071 7713 raft_consensus.cc:1261] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Refusing update from remote peer 4d762d9cd479416da10b933a0a94eacf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:26.359344 7868 consensus_queue.cc:1035] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa651c62fe848f39e269be2fdab4f57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:26.359623 7872 consensus_queue.cc:1035] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f05f49e630b472db248ebaa113915f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:26.360697 7759 heartbeater.cc:503] Master 127.0.98.60:43387 was elected leader, sending a full tablet report...
I20260620 09:00:26.361183 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.360724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:54405)
0620 09:00:26.361025 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.361093 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260620 09:00:26.361726 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.361461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:49641)
0620 09:00:26.361608 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.361659 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 09:00:26.364701 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.364452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.3:36745)
0620 09:00:26.364565 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.364623 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:00:26.364923 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.364077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:35193)
0620 09:00:26.364784 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.364853 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643}
I20260620 09:00:26.366264 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.365468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.2:42555)
0620 09:00:26.366149 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.366199 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615}
I20260620 09:00:26.367336 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.367075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36813 (local address 127.0.98.1:44501)
0620 09:00:26.367213 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.367265 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 09:00:26.367419 7878 mvcc.cc:205] Tried to move back new op lower bound from 7298850923960750080 to 7298850923897688064. Current Snapshot: MvccSnapshot[applied={T|T < 7298850923960750080}]
I20260620 09:00:26.374995 7298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:26.375151 7298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:26.379207 7298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 060d38baef5d4da78af1530433048e73: Sending DeleteTablet for 3 replicas of tablet 54cd3a00030442798480ee6224e920b2
I20260620 09:00:26.380033 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cd3a00030442798480ee6224e920b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48142
I20260620 09:00:26.380678 7419 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cd3a00030442798480ee6224e920b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41142
I20260620 09:00:26.380803 7556 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cd3a00030442798480ee6224e920b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54678
I20260620 09:00:26.381078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 7266
I20260620 09:00:26.381748 7884 tablet_replica.cc:330] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: stopping tablet replica
I20260620 09:00:26.381946 7884 raft_consensus.cc:2229] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:26.382118 7884 raft_consensus.cc:2258] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:26.385210 7884 ts_tablet_manager.cc:1905] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:26.388700 7886 tablet_replica.cc:330] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: stopping tablet replica
I20260620 09:00:26.388818 7884 ts_tablet_manager.cc:1918] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:26.388917 7884 log.cc:1196] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/54cd3a00030442798480ee6224e920b2
I20260620 09:00:26.389047 7886 raft_consensus.cc:2229] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:26.389243 7884 ts_tablet_manager.cc:1939] T 54cd3a00030442798480ee6224e920b2 P 3f05f49e630b472db248ebaa113915f1: Deleting consensus metadata
I20260620 09:00:26.389511 7886 raft_consensus.cc:2258] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:26.390183 7886 ts_tablet_manager.cc:1905] T 54cd3a00030442798480ee6224e920b2 P 4d762d9cd479416da10b933a0a94eacf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:26.390277 7292 catalog_manager.cc:4841] TS 3f05f49e630b472db248ebaa113915f1 (127.0.98.3:40699): tablet 54cd3a00030442798480ee6224e920b2 (table test-table [id=e48be2095f9140c7803b7f8be3718283]) successfully deleted
I20260620 09:00:26.391268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 7493
W20260620 09:00:26.391374 7859 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:26.390994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40363 (local address 127.0.0.1:41158)
0620 09:00:26.391182 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.391234 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40363: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 09:00:26.391932 7292 catalog_manager.cc:4584] TS 1fa651c62fe848f39e269be2fdab4f57 (127.0.98.1:40363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54cd3a00030442798480ee6224e920b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40363: connect: Connection reset by peer (error 104)
I20260620 09:00:26.397482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 7630
I20260620 09:00:26.397763 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:26.397442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36283 (local address 127.0.0.1:54692)
0620 09:00:26.397604 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:26.397668 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 09:00:26.404315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 6985
I20260620 09:00:26.410230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 7094
2026-06-20T09:00:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7244 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:26 dist-test-slave-scsf krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:26 dist-test-slave-scsf krb5kdc[7889](info): set up 2 sockets
Jun 20 09:00:26 dist-test-slave-scsf krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260620 09:00:27.449113 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:27 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:00:27.569293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:27.679803 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:27.680158 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:27.680241 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:27.684350 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:27.684473 7902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:00:27.684535 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:27.684573 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:27.684613 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:27.684648 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:27.689988 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43699
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:27.691119 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:27.692153 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:27.700812 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:27.700798 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:27.700945 7902 server_base.cc:1034] running on GCE node
W20260620 09:00:27.700805 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:27.701635 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:27.702668 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:27.703830 7902 hybrid_clock.cc:648] HybridClock initialized: now 1781946027703800 us; error 43 us; skew 500 ppm
Jun 20 09:00:27 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:27.711619 7902 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:00:27.712672 7902 webserver.cc:460] Webserver started at http://127.0.98.62:42723/ using document root <none> and password file <none>
I20260620 09:00:27.713284 7902 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:27.713377 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:27.713609 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:27.715668 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "ef9d15a823104d818c3306cdc23dec06"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "ae718ec18856624b8f98ffd932d6013d"
server_key_iv: "0b90478636e7c3d24b23c7d80f436050"
server_key_version: "encryptionkey@0"
I20260620 09:00:27.716226 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef9d15a823104d818c3306cdc23dec06"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "ae718ec18856624b8f98ffd932d6013d"
server_key_iv: "0b90478636e7c3d24b23c7d80f436050"
server_key_version: "encryptionkey@0"
I20260620 09:00:27.719959 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 09:00:27.722391 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:27.723645 7902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:27.723790 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "ef9d15a823104d818c3306cdc23dec06"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "ae718ec18856624b8f98ffd932d6013d"
server_key_iv: "0b90478636e7c3d24b23c7d80f436050"
server_key_version: "encryptionkey@0"
I20260620 09:00:27.723902 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:27.751418 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:27.754554 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:27.754762 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:27.763069 7902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43699
I20260620 09:00:27.763084 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43699 every 8 connection(s)
I20260620 09:00:27.764043 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:27.765869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 7902
I20260620 09:00:27.765988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:27.766285 392 external_mini_cluster.cc:1428] Setting key 845ba4eba27c4861a5b2d5f318fc2b17
I20260620 09:00:27.767153 7980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:27.772737 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Bootstrap starting.
Jun 20 09:00:27 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:27.775416 7980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:27.776289 7980 log.cc:824] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:27.778752 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: No bootstrap required, opened a new log
I20260620 09:00:27.781394 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:27.767694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.98.62:43699)
0620 09:00:27.768154 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:27.768168 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:27.768205 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:00:27.768875 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:27.768893 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:27.769208 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:27.769537 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:27.770374 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:27.771354 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:27.772139 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:27.772690 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:27.772858 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:27.775578 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:27.775583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:27.775597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:27.775624 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:27.778169 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:27.778634 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:27.778640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:27.778646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:27.778787 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:27.779096 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:27.779101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:27.779102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:27.779604 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:27.779781 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:27.780043 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:00:27.780352 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":177,"threads_started":1}
I20260620 09:00:27.782761 7980 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } }
I20260620 09:00:27.783049 7980 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:27.783109 7980 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9d15a823104d818c3306cdc23dec06, State: Initialized, Role: FOLLOWER
I20260620 09:00:27.783620 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } }
I20260620 09:00:27.783802 7980 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:27.783874 7980 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:27.783970 7980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:27.785002 7980 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } }
I20260620 09:00:27.785362 7980 leader_election.cc:302] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9d15a823104d818c3306cdc23dec06; no voters:
I20260620 09:00:27.785671 7980 leader_election.cc:288] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:27.785800 7985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:27.786034 7985 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [term 1 LEADER]: Becoming Leader. State: Replica: ef9d15a823104d818c3306cdc23dec06, State: Running, Role: LEADER
I20260620 09:00:27.786386 7985 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } }
I20260620 09:00:27.786993 7980 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:27.788180 7987 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9d15a823104d818c3306cdc23dec06. Latest consensus state: current_term: 1 leader_uuid: "ef9d15a823104d818c3306cdc23dec06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } } }
I20260620 09:00:27.788193 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9d15a823104d818c3306cdc23dec06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d15a823104d818c3306cdc23dec06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43699 } } }
I20260620 09:00:27.788321 7987 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:27.788324 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:27.788794 7994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:27.791688 7994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:27.797600 7994 catalog_manager.cc:1347] Generated new cluster ID: d2ab5da2904f47c78ae303caecb949e3
I20260620 09:00:27.797674 7994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:27.807615 7994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:27.808938 7994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:27.817830 7994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Generated new TSK 0
I20260620 09:00:27.818717 7994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:00:27.851661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43699
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:27.969753 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:27.970037 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:27.970083 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:27.974291 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:27.974360 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:27.974400 8006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:00:27.974435 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:27.979871 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:27.980963 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:27.981945 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:27.991552 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:27.991559 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:27.991564 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:27.991897 8006 server_base.cc:1034] running on GCE node
I20260620 09:00:27.992357 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:27.993074 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:27.994240 8006 hybrid_clock.cc:648] HybridClock initialized: now 1781946027994226 us; error 33 us; skew 500 ppm
Jun 20 09:00:27 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:27.997011 8006 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:00:27.998006 8006 webserver.cc:460] Webserver started at http://127.0.98.1:39403/ using document root <none> and password file <none>
I20260620 09:00:27.998620 8006 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:27.998682 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:27.998916 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:28.001061 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "45cdcf889e5349d7a069f15bf0956e90"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "c50953ed58947db7739e5929e100fc94"
server_key_iv: "319eb7af2987829e329e128c6ebeef98"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.001639 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "45cdcf889e5349d7a069f15bf0956e90"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "c50953ed58947db7739e5929e100fc94"
server_key_iv: "319eb7af2987829e329e128c6ebeef98"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.005477 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:28.007956 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.009227 8006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:28.009361 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "45cdcf889e5349d7a069f15bf0956e90"
format_stamp: "Formatted at 2026-06-20 09:00:27 on dist-test-slave-scsf"
server_key: "c50953ed58947db7739e5929e100fc94"
server_key_iv: "319eb7af2987829e329e128c6ebeef98"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.009511 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:28.024159 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:28.027376 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:28.027585 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:28.029220 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:28.029299 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.029364 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:28.029407 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:28.040486 8006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36853
I20260620 09:00:28.040565 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36853 every 8 connection(s)
I20260620 09:00:28.041587 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:28.043833 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.030088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52573 (local address 127.0.98.62:43699)
0620 09:00:28.030264 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.030270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.031119 (+ 849us) server_negotiation.cc:407] Connection header received
0620 09:00:28.032175 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.032178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.032249 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.032341 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.033845 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.034501 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.036009 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.036326 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.036444 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.040079 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.040085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.040093 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.040151 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.041841 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.042448 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.042452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.042453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.042544 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.042881 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.042883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.042884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.043060 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.043142 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.043601 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.043706 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:00:28.044724 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.030494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.1:52573)
0620 09:00:28.030955 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.031005 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.031926 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.032744 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.032751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.033083 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.033693 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.033710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.034651 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.034655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.035843 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.035852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.036514 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.036518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.036710 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.038397 (+ 1687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.038421 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.039896 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.041957 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.041963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.041978 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.042326 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.042649 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.042651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.042653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.042786 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.043159 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.043164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.043400 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.043553 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.043797 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260620 09:00:28.046159 8137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43699
I20260620 09:00:28.046325 8137 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:28.046859 8137 heartbeater.cc:511] Master 127.0.98.62:43699 requested a full tablet report, sending...
I20260620 09:00:28.048432 7933 ts_manager.cc:194] Registered new tserver with Master: 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853)
I20260620 09:00:28.048687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8006
I20260620 09:00:28.048812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:28.049114 392 external_mini_cluster.cc:1428] Setting key ef2379c772be579d59b47303cb2ad6be
I20260620 09:00:28.050041 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52573
I20260620 09:00:28.056098 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.048310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39883 (local address 127.0.98.62:43699)
0620 09:00:28.048428 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.048435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.048588 (+ 153us) server_negotiation.cc:407] Connection header received
0620 09:00:28.048689 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.048692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.048743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.048836 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.049364 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.049998 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.051166 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.051489 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.051598 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.052922 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.052926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.052929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.052959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.054363 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.054901 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.054904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.054908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.055003 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.055309 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.055311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.055312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.055473 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.055549 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.055879 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.055978 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:00:28.056310 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.048171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.1:39883)
0620 09:00:28.048493 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.048508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.048610 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.048868 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.048871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.049100 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.049221 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.049233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.050124 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.050127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.051037 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.051051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.051623 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.051629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.051768 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.052341 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.052351 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.052785 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.054477 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.054483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.054488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.054772 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.055101 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.055106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.055109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.055215 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.055572 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.055578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.055699 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.055835 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.056016 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 09:00:28.077428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43699
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:28.188732 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:28.189076 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:28.189188 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:28.193418 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:28.193512 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:28.193580 8146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:00:28.193665 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:28.199188 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:28.200342 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:28.201467 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:28.211289 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:28.211298 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:28.211309 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:28.211506 8146 server_base.cc:1034] running on GCE node
I20260620 09:00:28.212044 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:28.212778 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:28.213951 8146 hybrid_clock.cc:648] HybridClock initialized: now 1781946028213926 us; error 40 us; skew 500 ppm
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:28.216795 8146 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:00:28.217671 8146 webserver.cc:460] Webserver started at http://127.0.98.2:35997/ using document root <none> and password file <none>
I20260620 09:00:28.218266 8146 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:28.218359 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:28.218626 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:28.220775 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b8857e738cb4cfcb3615e27beb6029f"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "6e2fb3b0db596fce7adbd47adf19c23e"
server_key_iv: "7c0cde0b4a9b509ee7925a86cb4b1bd7"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.221365 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b8857e738cb4cfcb3615e27beb6029f"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "6e2fb3b0db596fce7adbd47adf19c23e"
server_key_iv: "7c0cde0b4a9b509ee7925a86cb4b1bd7"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.225114 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:00:28.227895 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.229321 8146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:00:28.229455 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "6b8857e738cb4cfcb3615e27beb6029f"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "6e2fb3b0db596fce7adbd47adf19c23e"
server_key_iv: "7c0cde0b4a9b509ee7925a86cb4b1bd7"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.229583 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:28.259539 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:28.263118 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:28.263337 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:28.264938 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:28.265012 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.265108 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:28.265153 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:28.276022 8146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34441
I20260620 09:00:28.276067 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34441 every 8 connection(s)
I20260620 09:00:28.276988 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:28.279469 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.265776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41157 (local address 127.0.98.62:43699)
0620 09:00:28.265950 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.265957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.266805 (+ 848us) server_negotiation.cc:407] Connection header received
0620 09:00:28.267897 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.267902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.267970 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.268062 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.270001 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.270923 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.272369 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.272665 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.272817 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.275856 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.275860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.275863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.275894 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.277691 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.278196 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.278199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.278200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.278281 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.278514 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.278516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.278517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.278680 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.278757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.279224 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.279346 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 09:00:28.280296 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.266166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.2:41157)
0620 09:00:28.266633 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.266683 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.267630 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.268582 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.268593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.269026 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.269790 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.269809 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.271063 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.271066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.272229 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.272246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.272856 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.272861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.273065 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.273722 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.273745 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.275640 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.277829 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.277834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.277849 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.278118 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.278344 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.278346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.278348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.278460 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.278768 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.278773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.279017 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.279175 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.279443 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":137,"threads_started":1}
I20260620 09:00:28.281567 8277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43699
I20260620 09:00:28.281704 8277 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:28.282267 8277 heartbeater.cc:511] Master 127.0.98.62:43699 requested a full tablet report, sending...
I20260620 09:00:28.283408 7934 ts_manager.cc:194] Registered new tserver with Master: 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441)
I20260620 09:00:28.284145 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41157
I20260620 09:00:28.284168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8146
I20260620 09:00:28.284281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:28.284724 392 external_mini_cluster.cc:1428] Setting key 4405999af17345e450f1fe50f533e814
I20260620 09:00:28.292999 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.283907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35977 (local address 127.0.98.62:43699)
0620 09:00:28.284026 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.284031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.284377 (+ 346us) server_negotiation.cc:407] Connection header received
0620 09:00:28.284539 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.284541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.284577 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.284616 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.285257 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.285850 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.288196 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.288579 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.288738 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.289901 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.289905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.289908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.289930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.291329 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.291732 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.291736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.291737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.291824 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.292095 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.292098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.292100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.292314 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.292416 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.292698 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.292820 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:00:28.293010 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.283832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.2:35977)
0620 09:00:28.284278 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.284293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.284393 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.284750 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.284752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.284967 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.285115 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.285126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.287430 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.287432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.288062 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.288071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.288735 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.288738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.288831 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.289374 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.289387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.289777 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.291435 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.291439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.291441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.291632 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.291911 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.291916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.291918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.292001 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.292414 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.292417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.292532 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.292648 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.292793 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":1,"thread_start_us":261,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:00:28.311347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43699
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:28.425522 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:28.425858 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:28.425935 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:28.430284 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:28.430395 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:28.430473 8286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:00:28.430572 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:28.436297 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:28.437470 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:28.438542 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:28.448370 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:28.448392 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:28.448364 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:28.448736 8286 server_base.cc:1034] running on GCE node
I20260620 09:00:28.449208 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:28.449908 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:28.451102 8286 hybrid_clock.cc:648] HybridClock initialized: now 1781946028451085 us; error 29 us; skew 500 ppm
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:28.453888 8286 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:00:28.454768 8286 webserver.cc:460] Webserver started at http://127.0.98.3:35867/ using document root <none> and password file <none>
I20260620 09:00:28.455332 8286 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:28.455421 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:28.455653 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:28.457835 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "a066652199de4e518a60ddff0df8be6c"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "d328357813de6ef50cb0248fb54be6bd"
server_key_iv: "955c6eb67f69afadcfbd63f93c387f1f"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.458420 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a066652199de4e518a60ddff0df8be6c"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "d328357813de6ef50cb0248fb54be6bd"
server_key_iv: "955c6eb67f69afadcfbd63f93c387f1f"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.462273 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:28.464757 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.465983 8286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:00:28.466127 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "a066652199de4e518a60ddff0df8be6c"
format_stamp: "Formatted at 2026-06-20 09:00:28 on dist-test-slave-scsf"
server_key: "d328357813de6ef50cb0248fb54be6bd"
server_key_iv: "955c6eb67f69afadcfbd63f93c387f1f"
server_key_version: "encryptionkey@0"
I20260620 09:00:28.466248 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:28.480881 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:28.484122 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:28.484342 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:28.485853 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:28.485930 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:28.486002 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:28.486045 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:28.496902 8286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44545
I20260620 09:00:28.496968 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44545 every 8 connection(s)
I20260620 09:00:28.498065 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:28.498565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8286
I20260620 09:00:28.498709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:28.499053 392 external_mini_cluster.cc:1428] Setting key f9021f5239f444df269a0ea59f61cc97
I20260620 09:00:28.501813 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.486756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52217 (local address 127.0.98.62:43699)
0620 09:00:28.486904 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.486909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.487846 (+ 937us) server_negotiation.cc:407] Connection header received
0620 09:00:28.488985 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.488991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.489052 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.489118 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.490903 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.491786 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.493828 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.494046 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.494139 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.497359 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.497363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.497366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.497396 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.499674 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.500191 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.500195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.500196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.500297 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.500733 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.500736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.500738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.500932 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.501042 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.501535 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.501658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:28.502619 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.487205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.3:52217)
0620 09:00:28.487694 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.487744 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.488721 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.489607 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.489615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.490043 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.490755 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.490778 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.491951 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.491956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.493703 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.493717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.494245 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.494251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.494468 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.495193 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.495231 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.497178 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.499776 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.499783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.499805 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.500075 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.500402 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.500405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.500407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.500626 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.501038 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.501045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.501299 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.501480 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.501811 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":141,"threads_started":1}
I20260620 09:00:28.503749 8417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43699
I20260620 09:00:28.503897 8417 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:28.504426 8417 heartbeater.cc:511] Master 127.0.98.62:43699 requested a full tablet report, sending...
I20260620 09:00:28.505493 7934 ts_manager.cc:194] Registered new tserver with Master: a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:28.506251 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52217
I20260620 09:00:28.513044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:00:28.514024 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.506016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52403 (local address 127.0.98.62:43699)
0620 09:00:28.506129 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.506133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.506732 (+ 599us) server_negotiation.cc:407] Connection header received
0620 09:00:28.506869 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.506872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.506920 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.507013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.507546 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.508271 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.509128 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.509335 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.509416 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.510515 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.510518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.510520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.510536 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.512075 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.512584 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.512588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.512589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.512683 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.513028 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.513034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.513036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.513329 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.513437 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.513747 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.513853 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:00:28.514056 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.505940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.98.3:52403)
0620 09:00:28.506637 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.506650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.506751 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.507042 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.507044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.507275 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.507415 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.507423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.508403 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.508405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.509022 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.509030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.509433 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.509436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.509500 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.510045 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.510056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.510402 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.512192 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.512196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.512198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.512444 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.512805 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.512811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.512816 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.512922 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.513426 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.513429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.513558 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.513685 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.513840 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":94,"threads_started":1}
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:00:28.537214 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:28.530920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.98.62:43699)
0620 09:00:28.531045 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.531049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.531062 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:28.531113 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.531116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.531154 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.531237 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.531934 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.532512 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.533421 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.533634 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.533715 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.537030 (+ 3315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35918: BlockingRecv error: recv got EOF from 127.0.0.1:35918 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:28.546555 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.538631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.98.62:43699)
0620 09:00:28.538784 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.538789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.538921 (+ 132us) server_negotiation.cc:407] Connection header received
0620 09:00:28.539074 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.539082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.539121 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.539211 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.539796 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.540299 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.541169 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.541391 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.541502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.543751 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.543754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.543756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.543779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.545104 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.545508 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.545511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.545512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.545580 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.545799 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.545801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.545802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.545980 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.546128 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.546344 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.546438 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:00:28.549479 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:28.552578 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:28.566422 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.561145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.98.1:36853)
0620 09:00:28.561425 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.561432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.561519 (+ 87us) server_negotiation.cc:407] Connection header received
0620 09:00:28.561774 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.561781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.562007 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.562160 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.562663 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.563765 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.565230 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.565777 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.565897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.566046 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.566189 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.566281 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260620 09:00:28.566422 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.561021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.0.1:56786)
0620 09:00:28.561371 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.561427 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.561590 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.562351 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.562359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.562387 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.562527 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.562535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.563862 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.563868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.565113 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.565121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.565895 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.565899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.566003 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.566037 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.566152 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.566272 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260620 09:00:28.567692 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.562040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.0.1:51300)
0620 09:00:28.562952 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.562966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.563057 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.563482 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.563485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.563498 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.563566 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.563571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.564996 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.564999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.566087 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.566095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.567310 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.567313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.567377 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.567394 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.567482 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.567551 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":89,"threads_started":1}
I20260620 09:00:28.567783 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.562561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.98.3:44545)
0620 09:00:28.562886 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.562897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.563052 (+ 155us) server_negotiation.cc:407] Connection header received
0620 09:00:28.563159 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.563167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.563360 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.563517 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.563635 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.564883 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.566207 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.567186 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.567306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.567484 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.567537 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.567625 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
I20260620 09:00:28.568171 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.561763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.0.1:38476)
0620 09:00:28.562260 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.562273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.562383 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.563228 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.563231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.563246 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.563322 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.563328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.564738 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.564742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.565940 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.565950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.567860 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.567867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.567959 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.567981 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.568035 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.568083 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":91,"threads_started":1}
I20260620 09:00:28.568912 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.562458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.0.98.2:34441)
0620 09:00:28.562764 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.562775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.562799 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:00:28.562870 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.562875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.563099 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.563252 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.563415 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.564595 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.566092 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.567704 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.568378 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.568612 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.568676 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.568776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260620 09:00:28.570602 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 628a19b75e934e97a6a0362dea599053 (DEFAULT_TABLE table=test-table [id=21fb493ab60e4bc199f24bbe452d8322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.570623 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 628a19b75e934e97a6a0362dea599053 (DEFAULT_TABLE table=test-table [id=21fb493ab60e4bc199f24bbe452d8322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.571789 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628a19b75e934e97a6a0362dea599053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.571775 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628a19b75e934e97a6a0362dea599053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.571513 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 628a19b75e934e97a6a0362dea599053 (DEFAULT_TABLE table=test-table [id=21fb493ab60e4bc199f24bbe452d8322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.572454 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628a19b75e934e97a6a0362dea599053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.579137 8445 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Bootstrap starting.
I20260620 09:00:28.579880 8447 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Bootstrap starting.
I20260620 09:00:28.580613 8446 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Bootstrap starting.
I20260620 09:00:28.581600 8445 tablet_bootstrap.cc:654] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.581765 8447 tablet_bootstrap.cc:654] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.582469 8447 log.cc:824] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:28.582463 8445 log.cc:824] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:28.582602 8446 tablet_bootstrap.cc:654] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.583367 8446 log.cc:824] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:28.584546 8447 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: No bootstrap required, opened a new log
I20260620 09:00:28.584759 8445 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: No bootstrap required, opened a new log
I20260620 09:00:28.584751 8447 ts_tablet_manager.cc:1397] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:00:28.584920 8445 ts_tablet_manager.cc:1397] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 09:00:28.585314 8446 tablet_bootstrap.cc:492] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: No bootstrap required, opened a new log
I20260620 09:00:28.585511 8446 ts_tablet_manager.cc:1397] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:00:28.588860 8445 raft_consensus.cc:348] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.588860 8447 raft_consensus.cc:348] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.589140 8447 raft_consensus.cc:374] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.589140 8445 raft_consensus.cc:374] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.589205 8445 raft_consensus.cc:729] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.589205 8447 raft_consensus.cc:729] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.589398 8446 raft_consensus.cc:348] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.589706 8446 raft_consensus.cc:374] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.589771 8446 raft_consensus.cc:729] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.589700 8445 consensus_queue.cc:260] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.589732 8447 consensus_queue.cc:260] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.590238 8446 consensus_queue.cc:260] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.590730 8447 ts_tablet_manager.cc:1428] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:00:28.590730 8445 ts_tablet_manager.cc:1428] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 09:00:28.591074 8277 heartbeater.cc:503] Master 127.0.98.62:43699 was elected leader, sending a full tablet report...
I20260620 09:00:28.591074 8417 heartbeater.cc:503] Master 127.0.98.62:43699 was elected leader, sending a full tablet report...
I20260620 09:00:28.591179 8446 ts_tablet_manager.cc:1428] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:00:28.592092 8137 heartbeater.cc:503] Master 127.0.98.62:43699 was elected leader, sending a full tablet report...
I20260620 09:00:28.713513 8451 raft_consensus.cc:482] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:28.713788 8451 raft_consensus.cc:504] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.714944 8451 leader_election.cc:288] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:28.718478 8452 raft_consensus.cc:482] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:28.718655 8452 raft_consensus.cc:504] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.719514 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.715194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.2:59033)
0620 09:00:28.715340 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.715357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.715453 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.715744 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.715747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.715768 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.715908 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.715913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.716717 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.716724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.717929 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.717939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.719081 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.719084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.719200 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.719226 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.719310 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.719387 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 09:00:28.719545 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.715303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59033 (local address 127.0.98.1:36853)
0620 09:00:28.715437 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.715440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.715473 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:00:28.715538 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.715543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.715595 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.715729 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.716029 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.716613 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.718079 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.718941 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.719050 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.719129 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.719344 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.719422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:00:28.720079 8452 leader_election.cc:288] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441)
I20260620 09:00:28.720335 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90" is_pre_election: true
I20260620 09:00:28.720511 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.715658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37311 (local address 127.0.98.3:44545)
0620 09:00:28.715817 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.715821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.715973 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:00:28.716124 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.716127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.716175 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.716260 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.716553 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.717216 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.719218 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.719813 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.719905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.719964 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.720292 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.720368 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:00:28.720697 8091 raft_consensus.cc:2454] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 0.
I20260620 09:00:28.720968 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.715515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.98.2:37311)
0620 09:00:28.715884 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.715894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.716011 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.716270 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.716273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.716288 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.716424 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.716428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.717317 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.717353 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.719086 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.719095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.720036 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.720042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.720155 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.720180 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.720727 (+ 547us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.720807 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260620 09:00:28.721274 8162 leader_election.cc:302] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f; no voters:
I20260620 09:00:28.721870 8451 raft_consensus.cc:2781] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:28.721938 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c" is_pre_election: true
I20260620 09:00:28.721977 8451 raft_consensus.cc:482] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:28.722123 8451 raft_consensus.cc:3037] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.722128 8371 raft_consensus.cc:2454] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 0.
I20260620 09:00:28.723539 8451 raft_consensus.cc:504] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.723905 8451 leader_election.cc:288] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 election: Requested vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:28.724320 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90"
I20260620 09:00:28.724480 8091 raft_consensus.cc:3037] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.724725 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.720218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.3:56255)
0620 09:00:28.720320 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.720335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.720409 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.721487 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.721491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.721509 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.721627 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.721634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.722526 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.722529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.723337 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.723346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.724251 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.724255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.724370 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.724392 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.724510 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.724606 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":13}
I20260620 09:00:28.725106 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.720981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56255 (local address 127.0.98.1:36853)
0620 09:00:28.721307 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.721312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.721325 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:28.721359 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.721361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.721399 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.721471 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.721746 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.722408 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.723465 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.724137 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.724248 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.724300 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.724890 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.724971 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260620 09:00:28.725836 8091 raft_consensus.cc:2454] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 1.
I20260620 09:00:28.725993 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.720485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.3:44321)
0620 09:00:28.720862 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.720876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.720965 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.721739 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.721741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.721756 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.722461 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.722471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.723394 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.723399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.724585 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.724595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.725531 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.725534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.725666 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.725704 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.725787 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.725862 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":162,"threads_started":1}
I20260620 09:00:28.726011 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.721166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44321 (local address 127.0.98.2:34441)
0620 09:00:28.721283 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.721287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.721301 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:28.721346 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.721348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.721393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.721471 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.722592 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.723256 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.724725 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.725287 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.725365 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.725420 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.725811 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.725871 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:00:28.726094 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90" is_pre_election: true
I20260620 09:00:28.726269 8090 raft_consensus.cc:2379] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a066652199de4e518a60ddff0df8be6c in current term 1: Already voted for candidate 6b8857e738cb4cfcb3615e27beb6029f in this term.
I20260620 09:00:28.726321 8162 leader_election.cc:302] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f; no voters:
I20260620 09:00:28.726483 8451 raft_consensus.cc:2781] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:28.726660 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c"
I20260620 09:00:28.726712 8451 raft_consensus.cc:686] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 LEADER]: Becoming Leader. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Running, Role: LEADER
I20260620 09:00:28.726787 8371 raft_consensus.cc:3037] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.727066 8451 consensus_queue.cc:237] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.727207 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628a19b75e934e97a6a0362dea599053" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f" is_pre_election: true
I20260620 09:00:28.728024 8371 raft_consensus.cc:2454] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 1.
I20260620 09:00:28.728019 8302 leader_election.cc:302] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a066652199de4e518a60ddff0df8be6c; no voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f
I20260620 09:00:28.728338 8452 raft_consensus.cc:2726] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 09:00:28.730731 7932 catalog_manager.cc:5495] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
W20260620 09:00:28.749279 8418 tablet.cc:2369] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:28.756790 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.753557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.0.98.2:34441)
0620 09:00:28.753691 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.753695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.753708 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:28.753833 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.753836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.753890 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.753980 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:28.754186 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.754694 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.755502 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.755705 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.755777 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.755993 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:28.756426 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:28.756573 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.756627 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.756679 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:00:28.762820 8090 raft_consensus.cc:1261] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Refusing update from remote peer 6b8857e738cb4cfcb3615e27beb6029f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:28.762934 8371 raft_consensus.cc:1261] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Refusing update from remote peer 6b8857e738cb4cfcb3615e27beb6029f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:28.763533 8451 consensus_queue.cc:1035] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:28.763728 8456 consensus_queue.cc:1035] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:28.772326 8459 mvcc.cc:205] Tried to move back new op lower bound from 7298850933804978176 to 7298850933670952960. Current Snapshot: MvccSnapshot[applied={T|T < 7298850933804978176}]
I20260620 09:00:28.773340 8460 mvcc.cc:205] Tried to move back new op lower bound from 7298850933804978176 to 7298850933670952960. Current Snapshot: MvccSnapshot[applied={T|T < 7298850933804978176}]
I20260620 09:00:28.773960 8461 mvcc.cc:205] Tried to move back new op lower bound from 7298850933804978176 to 7298850933670952960. Current Snapshot: MvccSnapshot[applied={T|T < 7298850933804978176}]
I20260620 09:00:28.783205 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:28.783361 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:28.785781 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Sending DeleteTablet for 3 replicas of tablet 628a19b75e934e97a6a0362dea599053
I20260620 09:00:28.786693 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 628a19b75e934e97a6a0362dea599053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260620 09:00:28.786711 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 628a19b75e934e97a6a0362dea599053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56786
I20260620 09:00:28.787240 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 628a19b75e934e97a6a0362dea599053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51300
I20260620 09:00:28.787408 8470 tablet_replica.cc:330] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: stopping tablet replica
I20260620 09:00:28.787350 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:00:28.787716 8470 raft_consensus.cc:2229] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 LEADER]: Raft consensus shutting down.
W20260620 09:00:28.787763 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:28.788151 8469 tablet_replica.cc:330] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: stopping tablet replica
I20260620 09:00:28.788195 8470 raft_consensus.cc:2258] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:28.788625 8469 raft_consensus.cc:2229] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:28.789225 8469 raft_consensus.cc:2258] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:28.789681 8470 ts_tablet_manager.cc:1905] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:28.789929 8471 tablet_replica.cc:330] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: stopping tablet replica
I20260620 09:00:28.790258 8471 raft_consensus.cc:2229] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:28.790601 8471 raft_consensus.cc:2258] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:28.791805 8469 ts_tablet_manager.cc:1905] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:28.792766 8470 ts_tablet_manager.cc:1918] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:28.792874 8470 log.cc:1196] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/628a19b75e934e97a6a0362dea599053
I20260620 09:00:28.793948 8071 tablet_service.cc:1467] Processing CreateTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 (DEFAULT_TABLE table=test-table [id=2e8b279f73094364b7b4323f3878a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.794260 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35fcaba40e4dd5beaeb132d2e73594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.795240 8469 ts_tablet_manager.cc:1918] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:28.795292 8471 ts_tablet_manager.cc:1905] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:28.795442 8469 log.cc:1196] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/628a19b75e934e97a6a0362dea599053
I20260620 09:00:28.795719 8351 tablet_service.cc:1467] Processing CreateTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 (DEFAULT_TABLE table=test-table [id=2e8b279f73094364b7b4323f3878a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.795903 8469 ts_tablet_manager.cc:1939] T 628a19b75e934e97a6a0362dea599053 P 45cdcf889e5349d7a069f15bf0956e90: Deleting consensus metadata
I20260620 09:00:28.796005 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35fcaba40e4dd5beaeb132d2e73594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.796689 8446 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Bootstrap starting.
I20260620 09:00:28.797724 8211 tablet_service.cc:1467] Processing CreateTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 (DEFAULT_TABLE table=test-table [id=2e8b279f73094364b7b4323f3878a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:28.797851 8446 tablet_bootstrap.cc:654] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.797977 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb35fcaba40e4dd5beaeb132d2e73594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.798326 8471 ts_tablet_manager.cc:1918] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:28.798401 8447 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Bootstrap starting.
I20260620 09:00:28.798436 8471 log.cc:1196] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/628a19b75e934e97a6a0362dea599053
I20260620 09:00:28.798761 8471 ts_tablet_manager.cc:1939] T 628a19b75e934e97a6a0362dea599053 P a066652199de4e518a60ddff0df8be6c: Deleting consensus metadata
I20260620 09:00:28.798894 8470 ts_tablet_manager.cc:1939] T 628a19b75e934e97a6a0362dea599053 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting consensus metadata
I20260620 09:00:28.799265 7918 catalog_manager.cc:4841] TS 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853): tablet 628a19b75e934e97a6a0362dea599053 (table test-table [id=21fb493ab60e4bc199f24bbe452d8322]) successfully deleted
I20260620 09:00:28.799443 8447 tablet_bootstrap.cc:654] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.800673 8447 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: No bootstrap required, opened a new log
I20260620 09:00:28.800683 7918 catalog_manager.cc:4841] TS 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441): tablet 628a19b75e934e97a6a0362dea599053 (table test-table [id=21fb493ab60e4bc199f24bbe452d8322]) successfully deleted
I20260620 09:00:28.800761 8447 ts_tablet_manager.cc:1397] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:28.800876 7918 catalog_manager.cc:4841] TS a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545): tablet 628a19b75e934e97a6a0362dea599053 (table test-table [id=21fb493ab60e4bc199f24bbe452d8322]) successfully deleted
I20260620 09:00:28.801218 8447 raft_consensus.cc:348] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.801337 8447 raft_consensus.cc:374] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.801383 8447 raft_consensus.cc:729] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.801506 8447 consensus_queue.cc:260] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.801818 8447 ts_tablet_manager.cc:1428] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:28.801930 8446 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: No bootstrap required, opened a new log
I20260620 09:00:28.802047 8446 ts_tablet_manager.cc:1397] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 09:00:28.802502 8446 raft_consensus.cc:348] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.802626 8446 raft_consensus.cc:374] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.802671 8446 raft_consensus.cc:729] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.802817 8445 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Bootstrap starting.
I20260620 09:00:28.802822 8446 consensus_queue.cc:260] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.803112 8446 ts_tablet_manager.cc:1428] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:28.803750 8445 tablet_bootstrap.cc:654] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.805653 8445 tablet_bootstrap.cc:492] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: No bootstrap required, opened a new log
I20260620 09:00:28.805743 8445 ts_tablet_manager.cc:1397] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 09:00:28.806149 8445 raft_consensus.cc:348] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.806267 8445 raft_consensus.cc:374] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.806298 8445 raft_consensus.cc:729] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.806411 8445 consensus_queue.cc:260] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.806646 8445 ts_tablet_manager.cc:1428] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:00:28.840374 8453 raft_consensus.cc:482] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:28.840597 8453 raft_consensus.cc:504] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.841871 8453 leader_election.cc:288] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:28.846894 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.842642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33775 (local address 127.0.98.3:44545)
0620 09:00:28.842767 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.842771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.842961 (+ 190us) server_negotiation.cc:407] Connection header received
0620 09:00:28.843117 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.843120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.843167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.843238 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.843572 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.844292 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.845679 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.846271 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.846376 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.846440 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.846662 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.846746 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:00:28.847081 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.842263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54439 (local address 127.0.98.2:34441)
0620 09:00:28.842405 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.842411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.842912 (+ 501us) server_negotiation.cc:407] Connection header received
0620 09:00:28.843056 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.843060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.843110 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.843203 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:28.843490 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.844276 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.845916 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.846498 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.846620 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.846680 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.846879 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.846945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:00:28.847205 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.842112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.1:54439)
0620 09:00:28.842812 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.842827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.842953 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.843235 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.843238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.843254 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.843372 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.843378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.844392 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.844396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.845778 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.845788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.846621 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.846624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.846751 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.846778 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.846852 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.846930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":78,"threads_started":1}
I20260620 09:00:28.847584 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.842544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.98.1:33775)
0620 09:00:28.842891 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.842904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.843023 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.843335 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.843338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.843357 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:28.843473 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.843479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.844528 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.844537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.845520 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.845529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.846401 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.846405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.846512 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.846537 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.847349 (+ 812us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.847428 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
I20260620 09:00:28.847754 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35fcaba40e4dd5beaeb132d2e73594" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f" is_pre_election: true
I20260620 09:00:28.847975 8231 raft_consensus.cc:2454] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 0.
I20260620 09:00:28.848371 8022 leader_election.cc:302] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f; no voters:
I20260620 09:00:28.848526 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35fcaba40e4dd5beaeb132d2e73594" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c" is_pre_election: true
I20260620 09:00:28.848668 8371 raft_consensus.cc:2454] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 0.
I20260620 09:00:28.848698 8453 raft_consensus.cc:2781] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:28.848757 8453 raft_consensus.cc:482] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:28.848794 8453 raft_consensus.cc:3037] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.850010 8453 raft_consensus.cc:504] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.850397 8453 leader_election.cc:288] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 election: Requested vote from peers 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:28.850788 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35fcaba40e4dd5beaeb132d2e73594" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c"
I20260620 09:00:28.850916 8371 raft_consensus.cc:3037] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.851058 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb35fcaba40e4dd5beaeb132d2e73594" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f"
I20260620 09:00:28.851226 8231 raft_consensus.cc:3037] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:28.852114 8371 raft_consensus.cc:2454] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 1.
I20260620 09:00:28.852391 8022 leader_election.cc:302] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, a066652199de4e518a60ddff0df8be6c; no voters:
I20260620 09:00:28.852427 8231 raft_consensus.cc:2454] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 1.
I20260620 09:00:28.852598 8453 raft_consensus.cc:2781] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:28.852800 8453 raft_consensus.cc:686] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 LEADER]: Becoming Leader. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Running, Role: LEADER
I20260620 09:00:28.853070 8453 consensus_queue.cc:237] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.855751 7932 catalog_manager.cc:5495] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "45cdcf889e5349d7a069f15bf0956e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:00:28 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946027, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:28.899509 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.890193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43699 (local address 127.0.0.1:35922)
0620 09:00:28.890512 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:28.890527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:28.890648 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:28.890930 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:28.890932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:28.891306 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:28.891404 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.891410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.892597 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.892599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.893246 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:28.893254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.893829 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.893831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:28.893891 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.894494 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:28.894506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:28.895906 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:28.897922 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.897932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.897938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.898170 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.898484 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:28.898486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:28.898488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:28.898566 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:28.899037 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:28.899045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:28.899165 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 09:00:28.899260 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 09:00:28.899375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
I20260620 09:00:28.899641 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:28.890308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.0.98.62:43699)
0620 09:00:28.890462 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:28.890467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:28.890609 (+ 142us) server_negotiation.cc:407] Connection header received
0620 09:00:28.890741 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:28.890744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:28.890796 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:28.890899 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:28.891534 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.892493 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.893369 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:28.893713 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:28.893822 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:28.896060 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:28.896064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:28.896066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:28.896092 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:28.897806 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.898276 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.898282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.898285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.898386 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:28.898657 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:28.898661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:28.898664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:28.898920 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:28.899017 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:28.899337 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 09:00:28.899462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:00:28.903723 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35922:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:00:28.904613 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:28.909933 8071 tablet_service.cc:1467] Processing CreateTablet for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b56631bfb6ec42f7ae73382d7b1322e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:28.909951 8351 tablet_service.cc:1467] Processing CreateTablet for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b56631bfb6ec42f7ae73382d7b1322e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:28.910045 8211 tablet_service.cc:1467] Processing CreateTablet for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b56631bfb6ec42f7ae73382d7b1322e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:00:28.910351 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.910351 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.910449 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5312f62c6b441a0a3b8b08f3f9c6ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:28.913364 8447 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: Bootstrap starting.
I20260620 09:00:28.913669 8446 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: Bootstrap starting.
I20260620 09:00:28.914429 8447 tablet_bootstrap.cc:654] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.914589 8446 tablet_bootstrap.cc:654] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.915485 8445 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: Bootstrap starting.
I20260620 09:00:28.916487 8445 tablet_bootstrap.cc:654] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:28.916500 8446 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: No bootstrap required, opened a new log
I20260620 09:00:28.916584 8446 ts_tablet_manager.cc:1397] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:28.917095 8447 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: No bootstrap required, opened a new log
I20260620 09:00:28.917042 8446 raft_consensus.cc:348] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.917167 8447 ts_tablet_manager.cc:1397] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 09:00:28.917166 8446 raft_consensus.cc:374] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.917198 8446 raft_consensus.cc:729] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.917325 8446 consensus_queue.cc:260] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.917588 8446 ts_tablet_manager.cc:1428] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:28.917671 8445 tablet_bootstrap.cc:492] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: No bootstrap required, opened a new log
I20260620 09:00:28.917608 8447 raft_consensus.cc:348] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.917719 8447 raft_consensus.cc:374] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.917745 8445 ts_tablet_manager.cc:1397] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:28.917758 8447 raft_consensus.cc:729] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.917903 8447 consensus_queue.cc:260] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.917898 8453 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:28.918025 8453 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:28.918184 8447 ts_tablet_manager.cc:1428] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:00:28.918203 8445 raft_consensus.cc:348] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.918309 8445 raft_consensus.cc:374] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:28.918341 8445 raft_consensus.cc:729] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Initialized, Role: FOLLOWER
I20260620 09:00:28.918465 8445 consensus_queue.cc:260] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:28.918745 8464 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:28.918911 8464 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:28.918852 8452 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:28.918975 8452 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:28.918751 8445 ts_tablet_manager.cc:1428] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 09:00:29.028923 8278 tablet.cc:2369] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:29.043311 8138 tablet.cc:2369] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:29.169716 8464 raft_consensus.cc:482] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:29.169859 8464 raft_consensus.cc:504] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.170297 8464 leader_election.cc:288] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:29.170854 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5312f62c6b441a0a3b8b08f3f9c6ffd" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90" is_pre_election: true
I20260620 09:00:29.170881 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5312f62c6b441a0a3b8b08f3f9c6ffd" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c" is_pre_election: true
I20260620 09:00:29.171042 8090 raft_consensus.cc:2454] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 0.
I20260620 09:00:29.171047 8371 raft_consensus.cc:2454] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 0.
I20260620 09:00:29.171381 8162 leader_election.cc:302] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8857e738cb4cfcb3615e27beb6029f, a066652199de4e518a60ddff0df8be6c; no voters:
I20260620 09:00:29.171551 8464 raft_consensus.cc:2781] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:29.171613 8464 raft_consensus.cc:482] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:29.171640 8464 raft_consensus.cc:3037] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.172852 8464 raft_consensus.cc:504] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.173194 8464 leader_election.cc:288] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 election: Requested vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:29.173628 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5312f62c6b441a0a3b8b08f3f9c6ffd" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c"
I20260620 09:00:29.173628 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5312f62c6b441a0a3b8b08f3f9c6ffd" candidate_uuid: "6b8857e738cb4cfcb3615e27beb6029f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90"
I20260620 09:00:29.173758 8090 raft_consensus.cc:3037] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.173758 8371 raft_consensus.cc:3037] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.174847 8090 raft_consensus.cc:2454] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 1.
I20260620 09:00:29.174847 8371 raft_consensus.cc:2454] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8857e738cb4cfcb3615e27beb6029f in term 1.
I20260620 09:00:29.175128 8162 leader_election.cc:302] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f; no voters:
I20260620 09:00:29.175282 8464 raft_consensus.cc:2781] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:29.175383 8464 raft_consensus.cc:686] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [term 1 LEADER]: Becoming Leader. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Running, Role: LEADER
I20260620 09:00:29.175576 8464 consensus_queue.cc:237] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.176216 8451 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: TxnStatusTablet state changed. Reason: New leader 6b8857e738cb4cfcb3615e27beb6029f. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.176327 8451 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:29.176609 8489 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:00:29.176786 8489 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:00:29.177392 7933 catalog_manager.cc:5495] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:29.218011 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.214554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.0.98.2:34441)
0620 09:00:29.214686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.214689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.214742 (+ 53us) server_negotiation.cc:407] Connection header received
0620 09:00:29.214929 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.214931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.214976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.215044 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.215308 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.215801 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.217015 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.217227 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.217305 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.217605 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.217679 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.217731 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.217854 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.217899 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:00:29.218060 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.214437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.0.1:38500)
0620 09:00:29.214682 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:29.214698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:29.214810 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:29.215089 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:29.215093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:29.215110 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:29.215218 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.215224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.215919 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.215922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.216890 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.216902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.217330 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.217334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.217420 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.217476 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.217756 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.217770 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:00:29.217836 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:00:29.217918 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 09:00:29.220739 8371 raft_consensus.cc:1261] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Refusing update from remote peer 6b8857e738cb4cfcb3615e27beb6029f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.220793 8090 raft_consensus.cc:1261] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Refusing update from remote peer 6b8857e738cb4cfcb3615e27beb6029f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.221221 8451 consensus_queue.cc:1035] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.221369 8464 consensus_queue.cc:1035] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.222846 8453 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: TxnStatusTablet state changed. Reason: New leader 6b8857e738cb4cfcb3615e27beb6029f. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.222962 8453 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:29.223407 8452 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: TxnStatusTablet state changed. Reason: New leader 6b8857e738cb4cfcb3615e27beb6029f. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.223498 8452 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:29.224982 8451 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.225093 8451 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:29.227312 8456 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.227423 8456 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 6b8857e738cb4cfcb3615e27beb6029f: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:00:29.227566 8453 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.227660 8453 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P 45cdcf889e5349d7a069f15bf0956e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:29.228791 8452 tablet_replica.cc:439] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8857e738cb4cfcb3615e27beb6029f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260620 09:00:29.228919 8452 tablet_replica.cc:442] T f5312f62c6b441a0a3b8b08f3f9c6ffd P a066652199de4e518a60ddff0df8be6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:00:29.235174 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.230580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.98.1:36853)
0620 09:00:29.231206 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.231211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.231223 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:00:29.231297 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.231299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.231357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.231469 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.231707 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.232501 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.233408 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.233754 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.233918 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.234171 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.234797 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.234914 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.234952 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.235007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":68,"threads_started":1}
I20260620 09:00:29.246222 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.242677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46105 (local address 127.0.98.2:34441)
0620 09:00:29.242806 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.242810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.243009 (+ 199us) server_negotiation.cc:407] Connection header received
0620 09:00:29.243166 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.243170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.243223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.243307 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.243553 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.244045 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.245106 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.245318 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.245426 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.245728 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.245810 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.245902 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.246061 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.246115 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:00:29.246302 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.242569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.1:46105)
0620 09:00:29.242925 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:29.242938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:29.243056 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:29.243318 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:29.243321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:29.243329 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:29.243441 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.243455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.244171 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.244175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.244985 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.244995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.245427 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.245430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.245524 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.245580 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.245944 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.245950 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:00:29.246029 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:00:29.246125 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260620 09:00:29.253628 8371 raft_consensus.cc:1261] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Refusing update from remote peer 45cdcf889e5349d7a069f15bf0956e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.253835 8231 raft_consensus.cc:1261] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Refusing update from remote peer 45cdcf889e5349d7a069f15bf0956e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.254103 8453 consensus_queue.cc:1035] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.254297 8477 consensus_queue.cc:1035] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.277889 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.273721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37805 (local address 127.0.98.1:36853)
0620 09:00:29.273849 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.273854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.274143 (+ 289us) server_negotiation.cc:407] Connection header received
0620 09:00:29.274268 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.274271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.274325 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.274409 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.274711 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.275447 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.276673 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.277012 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.277128 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.277434 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.277513 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.277579 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.277709 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.277765 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:00:29.278429 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.273581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.2:37805)
0620 09:00:29.274049 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:29.274064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:29.274167 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:29.274505 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:29.274508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:29.274514 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:00:29.274600 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.274605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.275573 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.275578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.276543 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:29.276554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.277151 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.277155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:29.277235 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.277284 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.277618 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.277624 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:00:29.278226 (+ 602us) client_negotiation.cc:240] Negotiation successful
0620 09:00:29.278306 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":125,"threads_started":1}
I20260620 09:00:29.314577 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.314761 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.317239 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Sending DeleteTablet for 3 replicas of tablet fb35fcaba40e4dd5beaeb132d2e73594
I20260620 09:00:29.317946 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260620 09:00:29.318117 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51300
I20260620 09:00:29.318168 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet fb35fcaba40e4dd5beaeb132d2e73594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56786
I20260620 09:00:29.318730 8505 tablet_replica.cc:330] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: stopping tablet replica
I20260620 09:00:29.318856 8505 raft_consensus.cc:2229] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.318962 8505 raft_consensus.cc:2258] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.319657 8505 ts_tablet_manager.cc:1905] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.322659 8505 ts_tablet_manager.cc:1918] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:29.322746 8505 log.cc:1196] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/fb35fcaba40e4dd5beaeb132d2e73594
I20260620 09:00:29.322997 8505 ts_tablet_manager.cc:1939] T fb35fcaba40e4dd5beaeb132d2e73594 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting consensus metadata
I20260620 09:00:29.323709 7918 catalog_manager.cc:4841] TS 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441): tablet fb35fcaba40e4dd5beaeb132d2e73594 (table test-table [id=2e8b279f73094364b7b4323f3878a6c4]) successfully deleted
I20260620 09:00:29.324632 8506 tablet_replica.cc:330] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: stopping tablet replica
I20260620 09:00:29.324795 8506 raft_consensus.cc:2229] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.324952 8506 raft_consensus.cc:2258] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.325249 8507 tablet_replica.cc:330] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: stopping tablet replica
I20260620 09:00:29.325389 8507 raft_consensus.cc:2229] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:29.325721 8507 raft_consensus.cc:2258] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.326298 8506 ts_tablet_manager.cc:1905] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.329437 8506 ts_tablet_manager.cc:1918] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:29.329543 8506 log.cc:1196] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/fb35fcaba40e4dd5beaeb132d2e73594
I20260620 09:00:29.329843 8506 ts_tablet_manager.cc:1939] T fb35fcaba40e4dd5beaeb132d2e73594 P a066652199de4e518a60ddff0df8be6c: Deleting consensus metadata
I20260620 09:00:29.330596 7918 catalog_manager.cc:4841] TS a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545): tablet fb35fcaba40e4dd5beaeb132d2e73594 (table test-table [id=2e8b279f73094364b7b4323f3878a6c4]) successfully deleted
I20260620 09:00:29.333077 8507 ts_tablet_manager.cc:1905] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.336119 8507 ts_tablet_manager.cc:1918] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:29.336210 8507 log.cc:1196] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/fb35fcaba40e4dd5beaeb132d2e73594
I20260620 09:00:29.336525 8507 ts_tablet_manager.cc:1939] T fb35fcaba40e4dd5beaeb132d2e73594 P 45cdcf889e5349d7a069f15bf0956e90: Deleting consensus metadata
I20260620 09:00:29.337306 7918 catalog_manager.cc:4841] TS 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853): tablet fb35fcaba40e4dd5beaeb132d2e73594 (table test-table [id=2e8b279f73094364b7b4323f3878a6c4]) successfully deleted
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:00:29.356665 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:29.349801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.0.98.62:43699)
0620 09:00:29.349945 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.349950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.350102 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:00:29.350225 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.350229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.350282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.350376 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:29.351053 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.351877 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.352863 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.353219 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.353327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.356522 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35936: BlockingRecv error: recv got EOF from 127.0.0.1:35936 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:00:29.367333 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.358369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.0.98.62:43699)
0620 09:00:29.358500 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.358505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.358739 (+ 234us) server_negotiation.cc:407] Connection header received
0620 09:00:29.358852 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.358854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.358894 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.358961 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:29.359694 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.360189 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.361351 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.361562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.361646 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.364185 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:29.364188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:29.364190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:29.364212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:29.365716 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:29.366144 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:29.366149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:29.366151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:29.366231 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:29.366499 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:29.366504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:29.366505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:29.366698 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:29.366835 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.367054 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.367169 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:00:29.370491 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:29.371017 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:29.375891 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (DEFAULT_TABLE table=test-table [id=014bdc090bd743b5a8f8b107ff220675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.375887 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (DEFAULT_TABLE table=test-table [id=014bdc090bd743b5a8f8b107ff220675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.376206 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.376206 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.378151 8447 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Bootstrap starting.
I20260620 09:00:29.378397 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (DEFAULT_TABLE table=test-table [id=014bdc090bd743b5a8f8b107ff220675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.378669 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.379041 8447 tablet_bootstrap.cc:654] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.380275 8447 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: No bootstrap required, opened a new log
I20260620 09:00:29.380364 8447 ts_tablet_manager.cc:1397] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:29.380656 8446 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Bootstrap starting.
I20260620 09:00:29.380892 8447 raft_consensus.cc:348] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.381000 8447 raft_consensus.cc:374] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.381045 8447 raft_consensus.cc:729] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.381170 8445 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Bootstrap starting.
I20260620 09:00:29.381170 8447 consensus_queue.cc:260] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.381428 8447 ts_tablet_manager.cc:1428] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:29.381603 8446 tablet_bootstrap.cc:654] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.381985 8445 tablet_bootstrap.cc:654] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.382831 8446 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: No bootstrap required, opened a new log
I20260620 09:00:29.382920 8446 ts_tablet_manager.cc:1397] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:29.383396 8446 raft_consensus.cc:348] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.383494 8445 tablet_bootstrap.cc:492] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: No bootstrap required, opened a new log
I20260620 09:00:29.383510 8446 raft_consensus.cc:374] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.383545 8446 raft_consensus.cc:729] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.383587 8445 ts_tablet_manager.cc:1397] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:29.383677 8446 consensus_queue.cc:260] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.383926 8446 ts_tablet_manager.cc:1428] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:29.383958 8445 raft_consensus.cc:348] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.384055 8445 raft_consensus.cc:374] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.384090 8445 raft_consensus.cc:729] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.384238 8445 consensus_queue.cc:260] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.384509 8445 ts_tablet_manager.cc:1428] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:29.429795 8452 raft_consensus.cc:482] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:29.429937 8452 raft_consensus.cc:504] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.430377 8452 leader_election.cc:288] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441)
I20260620 09:00:29.430912 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3cec5dc9a146bd824fe48e41bf0a5d" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f" is_pre_election: true
I20260620 09:00:29.430917 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3cec5dc9a146bd824fe48e41bf0a5d" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90" is_pre_election: true
I20260620 09:00:29.431082 8231 raft_consensus.cc:2454] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a066652199de4e518a60ddff0df8be6c in term 0.
I20260620 09:00:29.431085 8090 raft_consensus.cc:2454] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a066652199de4e518a60ddff0df8be6c in term 0.
I20260620 09:00:29.431406 8302 leader_election.cc:302] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, a066652199de4e518a60ddff0df8be6c; no voters:
I20260620 09:00:29.431591 8452 raft_consensus.cc:2781] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:29.431648 8452 raft_consensus.cc:482] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:29.431684 8452 raft_consensus.cc:3037] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.432847 8452 raft_consensus.cc:504] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.433285 8452 leader_election.cc:288] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 election: Requested vote from peers 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853), 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441)
I20260620 09:00:29.433759 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3cec5dc9a146bd824fe48e41bf0a5d" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f"
I20260620 09:00:29.433764 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3cec5dc9a146bd824fe48e41bf0a5d" candidate_uuid: "a066652199de4e518a60ddff0df8be6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cdcf889e5349d7a069f15bf0956e90"
I20260620 09:00:29.433895 8231 raft_consensus.cc:3037] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.433905 8090 raft_consensus.cc:3037] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.434849 8231 raft_consensus.cc:2454] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a066652199de4e518a60ddff0df8be6c in term 1.
I20260620 09:00:29.435154 8090 raft_consensus.cc:2454] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a066652199de4e518a60ddff0df8be6c in term 1.
I20260620 09:00:29.435173 8302 leader_election.cc:302] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8857e738cb4cfcb3615e27beb6029f, a066652199de4e518a60ddff0df8be6c; no voters:
I20260620 09:00:29.435345 8452 raft_consensus.cc:2781] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:29.435532 8452 raft_consensus.cc:686] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 1 LEADER]: Becoming Leader. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Running, Role: LEADER
I20260620 09:00:29.435789 8452 consensus_queue.cc:237] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.438148 7933 catalog_manager.cc:5495] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c reported cstate change: term changed from 0 to 1, leader changed from <none> to a066652199de4e518a60ddff0df8be6c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a066652199de4e518a60ddff0df8be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: HEALTHY } } }
I20260620 09:00:29.455482 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.451876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.98.3:44545)
0620 09:00:29.452163 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.452167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.452178 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:00:29.452243 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.452246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.452297 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.452430 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.452688 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.453230 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.454075 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.454313 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.454417 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.454683 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.455107 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.455235 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.455274 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.455324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
I20260620 09:00:29.458580 8090 raft_consensus.cc:1261] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Refusing update from remote peer a066652199de4e518a60ddff0df8be6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.458554 8231 raft_consensus.cc:1261] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Refusing update from remote peer a066652199de4e518a60ddff0df8be6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.459052 8523 consensus_queue.cc:1035] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.459331 8452 consensus_queue.cc:1035] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.472321 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.472529 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.474924 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Sending DeleteTablet for 3 replicas of tablet 0a3cec5dc9a146bd824fe48e41bf0a5d
I20260620 09:00:29.475684 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260620 09:00:29.475878 8505 tablet_replica.cc:330] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: stopping tablet replica
I20260620 09:00:29.475943 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56786
I20260620 09:00:29.476006 8505 raft_consensus.cc:2229] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.476118 8507 tablet_replica.cc:330] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: stopping tablet replica
I20260620 09:00:29.476143 8505 raft_consensus.cc:2258] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.476217 8507 raft_consensus.cc:2229] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.476336 8507 raft_consensus.cc:2258] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.476312 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a3cec5dc9a146bd824fe48e41bf0a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51300
I20260620 09:00:29.476513 8506 tablet_replica.cc:330] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: stopping tablet replica
I20260620 09:00:29.476617 8506 raft_consensus.cc:2229] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:29.476887 8506 raft_consensus.cc:2258] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.477308 8507 ts_tablet_manager.cc:1905] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.477332 8505 ts_tablet_manager.cc:1905] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.479694 8507 ts_tablet_manager.cc:1918] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:29.479759 8507 log.cc:1196] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/0a3cec5dc9a146bd824fe48e41bf0a5d
I20260620 09:00:29.479838 8505 ts_tablet_manager.cc:1918] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:29.479910 8505 log.cc:1196] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/0a3cec5dc9a146bd824fe48e41bf0a5d
I20260620 09:00:29.479993 8507 ts_tablet_manager.cc:1939] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 45cdcf889e5349d7a069f15bf0956e90: Deleting consensus metadata
I20260620 09:00:29.480162 8505 ts_tablet_manager.cc:1939] T 0a3cec5dc9a146bd824fe48e41bf0a5d P 6b8857e738cb4cfcb3615e27beb6029f: Deleting consensus metadata
I20260620 09:00:29.480671 7918 catalog_manager.cc:4841] TS 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853): tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (table test-table [id=014bdc090bd743b5a8f8b107ff220675]) successfully deleted
I20260620 09:00:29.480820 7918 catalog_manager.cc:4841] TS 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441): tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (table test-table [id=014bdc090bd743b5a8f8b107ff220675]) successfully deleted
I20260620 09:00:29.482065 8506 ts_tablet_manager.cc:1905] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.484405 8506 ts_tablet_manager.cc:1918] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:29.484505 8506 log.cc:1196] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/0a3cec5dc9a146bd824fe48e41bf0a5d
I20260620 09:00:29.484753 8506 ts_tablet_manager.cc:1939] T 0a3cec5dc9a146bd824fe48e41bf0a5d P a066652199de4e518a60ddff0df8be6c: Deleting consensus metadata
I20260620 09:00:29.484840 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:29.485283 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:29.485402 7918 catalog_manager.cc:4841] TS a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545): tablet 0a3cec5dc9a146bd824fe48e41bf0a5d (table test-table [id=014bdc090bd743b5a8f8b107ff220675]) successfully deleted
I20260620 09:00:29.490482 8211 tablet_service.cc:1467] Processing CreateTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 (DEFAULT_TABLE table=test-table [id=fa346fc73cb441aea65487749d55193d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.490787 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc46ddaf5434af294dc0a05c6756e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.490772 8071 tablet_service.cc:1467] Processing CreateTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 (DEFAULT_TABLE table=test-table [id=fa346fc73cb441aea65487749d55193d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.491024 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc46ddaf5434af294dc0a05c6756e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.493175 8445 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Bootstrap starting.
I20260620 09:00:29.493306 8446 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: Bootstrap starting.
I20260620 09:00:29.493989 8445 tablet_bootstrap.cc:654] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.494175 8446 tablet_bootstrap.cc:654] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.494671 8351 tablet_service.cc:1467] Processing CreateTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 (DEFAULT_TABLE table=test-table [id=fa346fc73cb441aea65487749d55193d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:29.494917 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc46ddaf5434af294dc0a05c6756e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:29.495752 8445 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: No bootstrap required, opened a new log
I20260620 09:00:29.495827 8445 ts_tablet_manager.cc:1397] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:00:29.496217 8445 raft_consensus.cc:348] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.496342 8445 raft_consensus.cc:374] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.496371 8445 raft_consensus.cc:729] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8857e738cb4cfcb3615e27beb6029f, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.496513 8445 consensus_queue.cc:260] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.496853 8445 ts_tablet_manager.cc:1428] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:29.498035 8447 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Bootstrap starting.
I20260620 09:00:29.498903 8447 tablet_bootstrap.cc:654] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:29.500964 8446 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: No bootstrap required, opened a new log
I20260620 09:00:29.501053 8446 ts_tablet_manager.cc:1397] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 09:00:29.501456 8446 raft_consensus.cc:348] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.501566 8446 raft_consensus.cc:374] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.501607 8446 raft_consensus.cc:729] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.501716 8446 consensus_queue.cc:260] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.501935 8446 ts_tablet_manager.cc:1428] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:00:29.509410 8447 tablet_bootstrap.cc:492] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: No bootstrap required, opened a new log
I20260620 09:00:29.509541 8447 ts_tablet_manager.cc:1397] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260620 09:00:29.510010 8447 raft_consensus.cc:348] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.510123 8447 raft_consensus.cc:374] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:29.510177 8447 raft_consensus.cc:729] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a066652199de4e518a60ddff0df8be6c, State: Initialized, Role: FOLLOWER
I20260620 09:00:29.510298 8447 consensus_queue.cc:260] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.510555 8447 ts_tablet_manager.cc:1428] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:00:29.822633 8502 raft_consensus.cc:482] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:29.822775 8502 raft_consensus.cc:504] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.823148 8502 leader_election.cc:288] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:29.823694 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc46ddaf5434af294dc0a05c6756e50" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f" is_pre_election: true
I20260620 09:00:29.823700 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc46ddaf5434af294dc0a05c6756e50" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c" is_pre_election: true
I20260620 09:00:29.823860 8231 raft_consensus.cc:2454] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 0.
I20260620 09:00:29.823874 8370 raft_consensus.cc:2454] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 0.
I20260620 09:00:29.824240 8022 leader_election.cc:302] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, a066652199de4e518a60ddff0df8be6c; no voters:
I20260620 09:00:29.824415 8502 raft_consensus.cc:2781] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:29.824496 8502 raft_consensus.cc:482] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:29.824537 8502 raft_consensus.cc:3037] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.825564 8502 raft_consensus.cc:504] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.825898 8502 leader_election.cc:288] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 election: Requested vote from peers 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441), a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545)
I20260620 09:00:29.826333 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc46ddaf5434af294dc0a05c6756e50" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a066652199de4e518a60ddff0df8be6c"
I20260620 09:00:29.826345 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc46ddaf5434af294dc0a05c6756e50" candidate_uuid: "45cdcf889e5349d7a069f15bf0956e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b8857e738cb4cfcb3615e27beb6029f"
I20260620 09:00:29.826465 8370 raft_consensus.cc:3037] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.826464 8231 raft_consensus.cc:3037] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:29.827601 8231 raft_consensus.cc:2454] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 1.
I20260620 09:00:29.827610 8370 raft_consensus.cc:2454] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cdcf889e5349d7a069f15bf0956e90 in term 1.
I20260620 09:00:29.827900 8022 leader_election.cc:302] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cdcf889e5349d7a069f15bf0956e90, 6b8857e738cb4cfcb3615e27beb6029f; no voters:
I20260620 09:00:29.828116 8502 raft_consensus.cc:2781] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:29.828337 8502 raft_consensus.cc:686] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 LEADER]: Becoming Leader. State: Replica: 45cdcf889e5349d7a069f15bf0956e90, State: Running, Role: LEADER
I20260620 09:00:29.828511 8502 consensus_queue.cc:237] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260620 09:00:29.830219 7932 catalog_manager.cc:5495] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "45cdcf889e5349d7a069f15bf0956e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cdcf889e5349d7a069f15bf0956e90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
I20260620 09:00:29.876688 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.874004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.1:36853)
0620 09:00:29.874245 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:29.874248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:29.874260 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:00:29.874298 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:29.874300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:29.874340 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:29.874411 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:29.874607 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.875087 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.875827 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:29.876054 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:29.876135 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:29.876333 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:29.876399 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:29.876485 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:29.876531 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 09:00:29.876574 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260620 09:00:29.884311 8231 raft_consensus.cc:1261] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Refusing update from remote peer 45cdcf889e5349d7a069f15bf0956e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.884311 8370 raft_consensus.cc:1261] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Refusing update from remote peer 45cdcf889e5349d7a069f15bf0956e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:29.884704 8533 consensus_queue.cc:1035] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8857e738cb4cfcb3615e27beb6029f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.884862 8502 consensus_queue.cc:1035] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a066652199de4e518a60ddff0df8be6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:29.915863 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.916016 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:29.918035 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9d15a823104d818c3306cdc23dec06: Sending DeleteTablet for 3 replicas of tablet bcc46ddaf5434af294dc0a05c6756e50
I20260620 09:00:29.918627 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56786
I20260620 09:00:29.918763 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51300
I20260620 09:00:29.918824 8507 tablet_replica.cc:330] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90: stopping tablet replica
I20260620 09:00:29.918964 8507 raft_consensus.cc:2229] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:29.919168 8507 raft_consensus.cc:2258] T bcc46ddaf5434af294dc0a05c6756e50 P 45cdcf889e5349d7a069f15bf0956e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.919206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8006
I20260620 09:00:29.919371 8506 tablet_replica.cc:330] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: stopping tablet replica
I20260620 09:00:29.919541 8506 raft_consensus.cc:2229] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.919660 8506 raft_consensus.cc:2258] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.920606 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc46ddaf5434af294dc0a05c6756e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38476
I20260620 09:00:29.920732 8506 ts_tablet_manager.cc:1905] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.920809 8505 tablet_replica.cc:330] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: stopping tablet replica
I20260620 09:00:29.920918 8505 raft_consensus.cc:2229] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:29.921039 8505 raft_consensus.cc:2258] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:29.921736 8505 ts_tablet_manager.cc:1905] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:29.923524 8506 ts_tablet_manager.cc:1918] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:29.923610 8506 log.cc:1196] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/bcc46ddaf5434af294dc0a05c6756e50
I20260620 09:00:29.923875 8506 ts_tablet_manager.cc:1939] T bcc46ddaf5434af294dc0a05c6756e50 P a066652199de4e518a60ddff0df8be6c: Deleting consensus metadata
I20260620 09:00:29.924327 8505 ts_tablet_manager.cc:1918] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:00:29.924407 8505 log.cc:1196] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/bcc46ddaf5434af294dc0a05c6756e50
I20260620 09:00:29.924718 8505 ts_tablet_manager.cc:1939] T bcc46ddaf5434af294dc0a05c6756e50 P 6b8857e738cb4cfcb3615e27beb6029f: Deleting consensus metadata
I20260620 09:00:29.924721 7918 catalog_manager.cc:4841] TS a066652199de4e518a60ddff0df8be6c (127.0.98.3:44545): tablet bcc46ddaf5434af294dc0a05c6756e50 (table test-table [id=fa346fc73cb441aea65487749d55193d]) successfully deleted
I20260620 09:00:29.925355 7918 catalog_manager.cc:4841] TS 6b8857e738cb4cfcb3615e27beb6029f (127.0.98.2:34441): tablet bcc46ddaf5434af294dc0a05c6756e50 (table test-table [id=fa346fc73cb441aea65487749d55193d]) successfully deleted
W20260620 09:00:29.928318 7918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36853 (error 108)
I20260620 09:00:29.928593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8146
I20260620 09:00:29.930718 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:29.930314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.0.1:56820)
0620 09:00:29.930535 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:29.930634 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
W20260620 09:00:29.930950 7918 catalog_manager.cc:4584] TS 45cdcf889e5349d7a069f15bf0956e90 (127.0.98.1:36853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcc46ddaf5434af294dc0a05c6756e50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36853: connect: Connection refused (error 111)
I20260620 09:00:29.936039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8286
I20260620 09:00:29.942556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 7902
2026-06-20T09:00:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3530 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[8547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[8547](info): set up 2 sockets
Jun 20 09:00:29 dist-test-slave-scsf krb5kdc[8547](info): commencing operation
krb5kdc: starting...
W20260620 09:00:30.977355 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:31 dist-test-slave-scsf krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:31.098975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:31.209635 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:31.209991 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:31.210086 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:31.214287 8560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:31.214378 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:31.214424 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:31.214488 8560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:31.214524 8560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:31.219805 8560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:31.221065 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:31.222184 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:31.230763 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:31.230762 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:31.230821 8560 server_base.cc:1034] running on GCE node
W20260620 09:00:31.230793 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:31.231607 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:31.232926 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:31.234081 8560 hybrid_clock.cc:648] HybridClock initialized: now 1781946031234050 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T09:00:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1293 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:31 dist-test-slave-scsf krb5kdc[8573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:31 dist-test-slave-scsf krb5kdc[8573](info): set up 2 sockets
Jun 20 09:00:31 dist-test-slave-scsf krb5kdc[8573](info): commencing operation
krb5kdc: starting...
W20260620 09:00:32.269292 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:32 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:32.391564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:32.502096 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:32.502445 8586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:32.502532 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:32.506804 8586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:32.506908 8586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:32.506946 8586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:32.507006 8586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:32.507046 8586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:32.512668 8586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:32.513840 8586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:32.514933 8586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:32.523769 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:32.523769 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:32.523835 8586 server_base.cc:1034] running on GCE node
W20260620 09:00:32.523772 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:32.524722 8586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:32.525869 8586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:32.527061 8586 hybrid_clock.cc:648] HybridClock initialized: now 1781946032527047 us; error 28 us; skew 500 ppm
Jun 20 09:00:32 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:32.530094 8586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:32.531034 8586 webserver.cc:460] Webserver started at http://127.0.98.62:42541/ using document root <none> and password file <none>
I20260620 09:00:32.531579 8586 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:32.531639 8586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:32.531806 8586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:32.533929 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "912f8a9f15bf4548863a3769dd633cff"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "83465b35b048d94f8dda2649435959bb"
server_key_iv: "5411835f35024ee7ac14049f10e04d72"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.534463 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "912f8a9f15bf4548863a3769dd633cff"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "83465b35b048d94f8dda2649435959bb"
server_key_iv: "5411835f35024ee7ac14049f10e04d72"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.538336 8586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:32.540763 8601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:32.542027 8586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:00:32.542158 8586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "912f8a9f15bf4548863a3769dd633cff"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "83465b35b048d94f8dda2649435959bb"
server_key_iv: "5411835f35024ee7ac14049f10e04d72"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.542269 8586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:32.562727 8586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:32.566176 8586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:32.566416 8586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:32.573839 8586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42923
I20260620 09:00:32.573872 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42923 every 8 connection(s)
I20260620 09:00:32.574827 8586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:32.577680 8654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:32.578086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8586
I20260620 09:00:32.578223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:32.578554 392 external_mini_cluster.cc:1428] Setting key a96c711f9a62f365a7f00c6369737391
I20260620 09:00:32.584839 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Bootstrap starting.
I20260620 09:00:32.587065 8654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:32.587813 8654 log.cc:824] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:00:32 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:32.589745 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: No bootstrap required, opened a new log
I20260620 09:00:32.593230 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:32.580029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.98.62:42923)
0620 09:00:32.580550 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:32.580566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:32.580601 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:00:32.581270 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:32.581286 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:32.581616 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:32.581968 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:32.582759 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.584065 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.585222 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.585715 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.585905 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:32.588512 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:32.588518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:32.588535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:32.588574 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:32.590333 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:32.590793 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:32.590800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:32.590812 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:32.590922 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:32.591211 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:32.591215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:32.591216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:32.591611 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:32.591806 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:32.592087 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 09:00:32.592411 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":154,"threads_started":1}
I20260620 09:00:32.593402 8654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:32.593669 8654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:32.593741 8654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f8a9f15bf4548863a3769dd633cff, State: Initialized, Role: FOLLOWER
I20260620 09:00:32.594179 8654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:32.594352 8654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:32.594434 8654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:32.594558 8654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:32.595543 8654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:32.595876 8654 leader_election.cc:302] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 912f8a9f15bf4548863a3769dd633cff; no voters:
I20260620 09:00:32.596176 8654 leader_election.cc:288] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:32.596297 8659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:32.596552 8659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 LEADER]: Becoming Leader. State: Replica: 912f8a9f15bf4548863a3769dd633cff, State: Running, Role: LEADER
I20260620 09:00:32.596900 8659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:32.597544 8654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:32.598739 8661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 912f8a9f15bf4548863a3769dd633cff. Latest consensus state: current_term: 1 leader_uuid: "912f8a9f15bf4548863a3769dd633cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 09:00:32.598905 8661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:32.598739 8660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "912f8a9f15bf4548863a3769dd633cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 09:00:32.598973 8660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:32.599282 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:32.601994 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:32.607544 8668 catalog_manager.cc:1347] Generated new cluster ID: ee038d94e9f24962904a683c48c31add
I20260620 09:00:32.607628 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:32.637365 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:32.638374 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:32.652060 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Generated new TSK 0
I20260620 09:00:32.652995 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:32.685209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:32.804486 8680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:32.804764 8680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:32.804805 8680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:32.808647 8680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:32.808702 8680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:32.808755 8680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:32.814000 8680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:32.815017 8680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:32.815976 8680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:32.825184 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:32.825176 8685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:32.825176 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:32.825359 8680 server_base.cc:1034] running on GCE node
I20260620 09:00:32.825930 8680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:32.826524 8680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:32.827688 8680 hybrid_clock.cc:648] HybridClock initialized: now 1781946032827666 us; error 33 us; skew 500 ppm
Jun 20 09:00:32 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:32.834161 8680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:32.835152 8680 webserver.cc:460] Webserver started at http://127.0.98.1:45811/ using document root <none> and password file <none>
I20260620 09:00:32.835721 8680 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:32.835783 8680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:32.835968 8680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:32.838208 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "74d613eaebe54b62b7792d209a70c85e"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "35c4eeae6eca608bab3cc15e18237a37"
server_key_iv: "839f743510008da1bac117b52f813910"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.838740 8680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74d613eaebe54b62b7792d209a70c85e"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "35c4eeae6eca608bab3cc15e18237a37"
server_key_iv: "839f743510008da1bac117b52f813910"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.842957 8680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:32.845429 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:32.846731 8680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:32.846859 8680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "74d613eaebe54b62b7792d209a70c85e"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "35c4eeae6eca608bab3cc15e18237a37"
server_key_iv: "839f743510008da1bac117b52f813910"
server_key_version: "encryptionkey@0"
I20260620 09:00:32.846966 8680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:32.861766 8680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:32.865253 8680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:32.865473 8680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:32.866111 8680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:32.866940 8680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:32.866988 8680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:32.867057 8680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:32.867089 8680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:32.876363 8680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45955
I20260620 09:00:32.876367 8808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45955 every 8 connection(s)
I20260620 09:00:32.877450 8680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:32.881911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8680
I20260620 09:00:32.882056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:32.882351 392 external_mini_cluster.cc:1428] Setting key 1feec48444e04aa18116eb743209501d
Jun 20 09:00:32 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:32.893146 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:32.879343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59185 (local address 127.0.98.62:42923)
0620 09:00:32.879490 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:32.879495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:32.880493 (+ 998us) server_negotiation.cc:407] Connection header received
0620 09:00:32.881882 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:32.881887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:32.881953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:32.882058 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:32.883452 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.884086 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.885809 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.886220 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.886339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:32.888974 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:32.888978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:32.888980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:32.889003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:32.890408 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:32.890982 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:32.890986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:32.890988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:32.891069 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:32.891526 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:32.891531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:32.891565 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:32.891800 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:32.891924 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:32.892771 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 09:00:32.892921 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:00:32.894186 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:32.879824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:59185)
0620 09:00:32.880305 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:32.880350 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:32.881665 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:32.882252 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:32.882259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:32.882697 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:32.883313 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:32.883329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.884235 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.884242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:32.885694 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:32.885704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:32.886368 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:32.886371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:32.886516 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:32.887131 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:32.887147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:32.888798 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:32.890544 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:32.890550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:32.890567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:32.890875 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:32.891171 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:32.891174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:32.891175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:32.891395 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:32.891984 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:32.891990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:32.892308 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 09:00:32.892580 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 09:00:32.893030 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:00:32.895562 8809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 09:00:32.895828 8809 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:32.896586 8809 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
I20260620 09:00:32.898489 8618 ts_manager.cc:194] Registered new tserver with Master: 74d613eaebe54b62b7792d209a70c85e (127.0.98.1:45955)
I20260620 09:00:32.899830 8618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:32.910909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:33.023838 8814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:33.024144 8814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:33.024228 8814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:33.028357 8814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:33.028462 8814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:33.028573 8814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:33.034014 8814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:33.035163 8814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:33.036231 8814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:33.046219 8820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.046227 8822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.046244 8819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:33.046494 8814 server_base.cc:1034] running on GCE node
I20260620 09:00:33.047127 8814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:33.047820 8814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:33.049005 8814 hybrid_clock.cc:648] HybridClock initialized: now 1781946033048990 us; error 35 us; skew 500 ppm
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:33.051918 8814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:33.052904 8814 webserver.cc:460] Webserver started at http://127.0.98.2:33831/ using document root <none> and password file <none>
I20260620 09:00:33.053489 8814 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:33.053582 8814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:33.053818 8814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:33.055899 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ae5504d1e914674abbb4fe3f0385079"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "e5697d6c929b089ae1663ed1ceee9f0a"
server_key_iv: "9a45ad636c7747cbc391017a0ea3eb04"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.056527 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ae5504d1e914674abbb4fe3f0385079"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "e5697d6c929b089ae1663ed1ceee9f0a"
server_key_iv: "9a45ad636c7747cbc391017a0ea3eb04"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.060161 8814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:00:33.062645 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.063859 8814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:33.064010 8814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "5ae5504d1e914674abbb4fe3f0385079"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "e5697d6c929b089ae1663ed1ceee9f0a"
server_key_iv: "9a45ad636c7747cbc391017a0ea3eb04"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.064126 8814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:33.085513 8814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:33.088502 8814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:33.088732 8814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:33.089347 8814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:33.090178 8814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:33.090250 8814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.090325 8814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:33.090369 8814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.099726 8814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36975
I20260620 09:00:33.099753 8942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36975 every 8 connection(s)
I20260620 09:00:33.100888 8814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:33.107854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8814
I20260620 09:00:33.107986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:33.108273 392 external_mini_cluster.cc:1428] Setting key cf435746b8b122b0cb4c14fbe4c4b520
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:33.116078 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.102731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40571 (local address 127.0.98.62:42923)
0620 09:00:33.102865 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:33.102868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:33.103697 (+ 829us) server_negotiation.cc:407] Connection header received
0620 09:00:33.105140 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:33.105144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:33.105202 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:33.105312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:33.106457 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.107046 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.108557 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.108841 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.108954 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.111891 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:33.111894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:33.111896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:33.111922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:33.113499 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:33.114162 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:33.114165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:33.114166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:33.114252 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:33.114768 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:33.114771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:33.114772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:33.114990 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:33.115118 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:33.115839 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 09:00:33.115954 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:00:33.117426 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.103115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:40571)
0620 09:00:33.103552 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:33.103593 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:33.104933 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:33.105489 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:33.105495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:33.105845 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:33.106314 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.106327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.107175 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.107185 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.108412 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.108422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.109000 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.109004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.109185 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.109796 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:33.109821 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:33.111698 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:33.113641 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:33.113649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:33.113669 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:33.114056 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:33.114361 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:33.114366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:33.114370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:33.114581 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:33.115140 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:33.115149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:33.115545 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 09:00:33.115784 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 09:00:33.116187 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:33.118685 8943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 09:00:33.118963 8943 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:33.119467 8943 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:00:33.120599 8618 ts_manager.cc:194] Registered new tserver with Master: 5ae5504d1e914674abbb4fe3f0385079 (127.0.98.2:36975)
I20260620 09:00:33.121448 8618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:33.135396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:33.248397 8948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:33.248747 8948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:33.248845 8948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:33.253189 8948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:33.253283 8948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:33.253389 8948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:33.259274 8948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:33.260438 8948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:33.261546 8948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:33.271411 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.271410 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.271410 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:33.271670 8948 server_base.cc:1034] running on GCE node
I20260620 09:00:33.272238 8948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:33.272980 8948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:33.274221 8948 hybrid_clock.cc:648] HybridClock initialized: now 1781946033274205 us; error 38 us; skew 500 ppm
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:33.277033 8948 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:33.277951 8948 webserver.cc:460] Webserver started at http://127.0.98.3:35877/ using document root <none> and password file <none>
I20260620 09:00:33.278538 8948 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:33.278632 8948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:33.278869 8948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:33.281009 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cc4a527e9be49739988fd5e1cf014c2"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "29e9aa91a4df27735ab14612c5ddcca0"
server_key_iv: "9e305c37def59de47990d355a1e5398f"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.281601 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cc4a527e9be49739988fd5e1cf014c2"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "29e9aa91a4df27735ab14612c5ddcca0"
server_key_iv: "9e305c37def59de47990d355a1e5398f"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.285295 8948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:00:33.288055 8963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.289261 8948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:00:33.289407 8948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "3cc4a527e9be49739988fd5e1cf014c2"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "29e9aa91a4df27735ab14612c5ddcca0"
server_key_iv: "9e305c37def59de47990d355a1e5398f"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.289525 8948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:33.322597 8948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:33.325765 8948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:33.325987 8948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:33.326609 8948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:33.327469 8948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:33.327543 8948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.327615 8948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:33.327669 8948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.337006 8948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43791
I20260620 09:00:33.337028 9076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43791 every 8 connection(s)
I20260620 09:00:33.338017 8948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:33.342111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 8948
I20260620 09:00:33.342252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:33.342581 392 external_mini_cluster.cc:1428] Setting key 03c380bb8ef50d59709b6c38eff7e68a
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:33.352557 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.339873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45397 (local address 127.0.98.62:42923)
0620 09:00:33.340044 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:33.340049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:33.340984 (+ 935us) server_negotiation.cc:407] Connection header received
0620 09:00:33.342594 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:33.342597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:33.342673 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:33.342764 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:33.344317 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.345007 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.346248 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.346484 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.346603 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.349150 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:33.349153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:33.349155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:33.349179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:33.350615 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:33.351098 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:33.351101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:33.351102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:33.351178 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:33.351478 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:33.351480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:33.351481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:33.351679 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:33.351793 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:33.352273 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 09:00:33.352375 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 09:00:33.353394 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.340315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:45397)
0620 09:00:33.340831 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:33.340873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:33.342348 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:33.342986 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:33.342994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:33.343513 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:33.344169 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.344184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.345121 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.345124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.346149 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.346158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.346643 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.346649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.346864 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.347477 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:33.347493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:33.348962 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:33.350728 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:33.350733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:33.350746 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:33.351010 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:33.351261 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:33.351263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:33.351264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:33.351396 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:33.351804 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:33.351811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:33.352065 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 09:00:33.352233 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 09:00:33.352561 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":182,"threads_started":1}
I20260620 09:00:33.354553 9077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 09:00:33.354773 9077 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:33.355309 9077 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
I20260620 09:00:33.356446 8617 ts_manager.cc:194] Registered new tserver with Master: 3cc4a527e9be49739988fd5e1cf014c2 (127.0.98.3:43791)
I20260620 09:00:33.357076 8617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45397
I20260620 09:00:33.368933 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:00:33.442484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8680
I20260620 09:00:33.448065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8814
I20260620 09:00:33.453567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8948
I20260620 09:00:33.458853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 8586
I20260620 09:00:33.465497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--webserver_interface=127.0.98.62
--webserver_port=42541
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:33.578540 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:33.578871 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:33.578953 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:33.583117 9086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:33.583186 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:33.583246 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:33.583276 9086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:33.583314 9086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:33.588778 9086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:33.589884 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:33.590957 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:33.599395 9092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.599401 9094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.599401 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:33.599752 9086 server_base.cc:1034] running on GCE node
I20260620 09:00:33.600374 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:33.601645 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:33.602820 9086 hybrid_clock.cc:648] HybridClock initialized: now 1781946033602810 us; error 38 us; skew 500 ppm
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:33.605706 9086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:33.606571 9086 webserver.cc:460] Webserver started at http://127.0.98.62:42541/ using document root <none> and password file <none>
I20260620 09:00:33.607101 9086 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:33.607172 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:33.611120 9086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:33.613314 9101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.614591 9086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:33.614724 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "912f8a9f15bf4548863a3769dd633cff"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "83465b35b048d94f8dda2649435959bb"
server_key_iv: "5411835f35024ee7ac14049f10e04d72"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.615283 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:33.629014 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:33.632432 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:33.632705 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:33.639912 9086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42923
I20260620 09:00:33.639930 9153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42923 every 8 connection(s)
I20260620 09:00:33.641115 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:33.641741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9086
I20260620 09:00:33.642177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45955
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42923
--webserver_port=45811
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:00:33.646132 9154 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:00:33.648950 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Bootstrap starting.
I20260620 09:00:33.672202 9154 log.cc:824] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:33.680431 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:00:33.680907 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Bootstrap complete.
I20260620 09:00:33.686506 9154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:33.687070 9154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f8a9f15bf4548863a3769dd633cff, State: Initialized, Role: FOLLOWER
I20260620 09:00:33.687668 9154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:33.687824 9154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:33.687896 9154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:33.687997 9154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:00:33.690713 9154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:33.691108 9154 leader_election.cc:302] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 912f8a9f15bf4548863a3769dd633cff; no voters:
I20260620 09:00:33.691493 9154 leader_election.cc:288] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:00:33.691648 9158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:00:33.691932 9158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [term 2 LEADER]: Becoming Leader. State: Replica: 912f8a9f15bf4548863a3769dd633cff, State: Running, Role: LEADER
I20260620 09:00:33.692266 9158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 09:00:33.692798 9154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:33.693672 9160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 912f8a9f15bf4548863a3769dd633cff. Latest consensus state: current_term: 2 leader_uuid: "912f8a9f15bf4548863a3769dd633cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 09:00:33.693795 9160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:33.694096 9159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "912f8a9f15bf4548863a3769dd633cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "912f8a9f15bf4548863a3769dd633cff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 09:00:33.694198 9159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:33.697929 9174 catalog_manager.cc:1259] Loaded cluster ID: ee038d94e9f24962904a683c48c31add
I20260620 09:00:33.697992 9174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: loading cluster ID for follower catalog manager: success
I20260620 09:00:33.700367 9174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: acquiring CA information for follower catalog manager: success
I20260620 09:00:33.701777 9174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:00:33.701936 9167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:33.702778 9167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:33.703194 9167 catalog_manager.cc:1259] Loaded cluster ID: ee038d94e9f24962904a683c48c31add
I20260620 09:00:33.703245 9167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:33.703948 9167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:33.704409 9167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 912f8a9f15bf4548863a3769dd633cff: Loaded TSK: 0
I20260620 09:00:33.705220 9167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:00:33.775712 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:33.775975 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:33.776017 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:33.779894 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:33.779965 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:33.780023 9156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:33.785313 9156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45811
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:33.786324 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:33.787283 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:33.796795 9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.796795 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.796846 9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:33.797217 9156 server_base.cc:1034] running on GCE node
I20260620 09:00:33.797628 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:33.798302 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:33.799487 9156 hybrid_clock.cc:648] HybridClock initialized: now 1781946033799474 us; error 37 us; skew 500 ppm
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:33.802434 9156 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:00:33.803452 9156 webserver.cc:460] Webserver started at http://127.0.98.1:45811/ using document root <none> and password file <none>
I20260620 09:00:33.804025 9156 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:33.804140 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:33.808243 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:33.810483 9190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.811836 9156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:00:33.811931 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "74d613eaebe54b62b7792d209a70c85e"
format_stamp: "Formatted at 2026-06-20 09:00:32 on dist-test-slave-scsf"
server_key: "35c4eeae6eca608bab3cc15e18237a37"
server_key_iv: "839f743510008da1bac117b52f813910"
server_key_version: "encryptionkey@0"
I20260620 09:00:33.812534 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:33.829818 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:33.832882 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:33.833050 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:33.833652 9156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:33.834540 9156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:33.834589 9156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.834628 9156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:33.834648 9156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:33.844159 9156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45955
I20260620 09:00:33.844182 9303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45955 every 8 connection(s)
I20260620 09:00:33.845206 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:33.851361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9156
I20260620 09:00:33.851858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36975
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42923
--webserver_port=33831
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:33.859076 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.847525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:56549)
0620 09:00:33.847981 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:33.848032 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:33.849390 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:33.850461 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:33.850468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:33.850830 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:33.851452 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.851488 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.852904 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.852911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.853891 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.853901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.855040 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.855042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.855179 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.855804 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:33.855820 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:33.858312 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":132,"threads_started":1}
W20260620 09:00:33.859660 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:33.847377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56549 (local address 127.0.98.62:42923)
0620 09:00:33.847816 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:33.847841 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:33.848153 (+ 312us) server_negotiation.cc:407] Connection header received
0620 09:00:33.849754 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:33.849772 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:33.850072 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:33.850350 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:33.851626 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.852758 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.854027 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.854880 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.855076 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.858637 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56549: BlockingRecv error: recv got EOF from 127.0.98.1:56549 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
W20260620 09:00:33.859771 9304 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42923 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42923: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:33.865468 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.860388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:58235)
0620 09:00:33.860501 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:33.860516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:33.860599 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:33.860842 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:33.860845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:33.861007 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:33.861095 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.861100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.861839 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.861845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.862384 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.862391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.862865 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.862868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.862966 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.863476 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:33.863488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:33.865320 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 09:00:33.865864 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:33.860493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58235 (local address 127.0.98.62:42923)
0620 09:00:33.860625 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:33.860632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:33.860648 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:33.860689 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:33.860692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:33.860743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:33.860830 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:33.861217 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.861729 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.862504 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.862744 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.862850 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.865439 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58235: BlockingRecv error: recv got EOF from 127.0.98.1:58235 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:33.871168 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:33.866055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:58623)
0620 09:00:33.866159 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:33.866173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:33.866252 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:33.866541 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:33.866544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:33.866700 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:33.866796 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.866801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.867535 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.867541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.868080 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:33.868087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.868554 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.868557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:33.868631 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.869096 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:33.869109 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:33.871039 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:00:33.871356 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:33.866236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58623 (local address 127.0.98.62:42923)
0620 09:00:33.866366 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:33.866375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:33.866385 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:00:33.866412 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:33.866413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:33.866446 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:33.866528 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:33.866907 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.867419 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.868203 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:33.868406 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:33.868552 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:33.871182 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58623: BlockingRecv error: recv got EOF from 127.0.98.1:58623 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 09:00:33.871404 9304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:00:33.967943 9308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:33.968215 9308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:33.968266 9308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:33.972294 9308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:33.972365 9308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:33.972486 9308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:33.977814 9308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33831
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:33.978912 9308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:33.979962 9308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:33.989400 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.989421 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:33.989400 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:33.989782 9308 server_base.cc:1034] running on GCE node
I20260620 09:00:33.990315 9308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:33.990970 9308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:33.992134 9308 hybrid_clock.cc:648] HybridClock initialized: now 1781946033992113 us; error 33 us; skew 500 ppm
Jun 20 09:00:33 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:33.995175 9308 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:00:33.996042 9308 webserver.cc:460] Webserver started at http://127.0.98.2:33831/ using document root <none> and password file <none>
I20260620 09:00:33.996688 9308 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:33.996770 9308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:34.000746 9308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:00:34.002849 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.004058 9308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:34.004185 9308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "5ae5504d1e914674abbb4fe3f0385079"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "e5697d6c929b089ae1663ed1ceee9f0a"
server_key_iv: "9a45ad636c7747cbc391017a0ea3eb04"
server_key_version: "encryptionkey@0"
I20260620 09:00:34.004802 9308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:34.017468 9308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:34.020913 9308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:34.021122 9308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:34.021739 9308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:34.022584 9308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:34.022632 9308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.022718 9308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:34.022751 9308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.031508 9308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36975
I20260620 09:00:34.031581 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36975 every 8 connection(s)
I20260620 09:00:34.032637 9308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:34.038647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9308
I20260620 09:00:34.039080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43791
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42923
--webserver_port=35877
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:00:34.044613 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.034480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54109 (local address 127.0.98.62:42923)
0620 09:00:34.034618 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.034622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.035500 (+ 878us) server_negotiation.cc:407] Connection header received
0620 09:00:34.036951 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.036954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.037017 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.037129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.038318 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.038927 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.040330 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.040613 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.040726 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.044402 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54109: BlockingRecv error: recv got EOF from 127.0.98.2:54109 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:00:34.045154 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.034893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:54109)
0620 09:00:34.035344 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.035386 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.036717 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.037313 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.037320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.037681 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.038168 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.038181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.039053 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.039056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.040198 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.040208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.040777 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.040781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.040969 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.041604 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.041620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.044189 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
W20260620 09:00:34.045830 9437 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42923 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42923: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.052073 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.046341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:53747)
0620 09:00:34.046461 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.046476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.046561 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.046846 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.046848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.046984 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.047077 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.047082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.048186 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.048188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.048792 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.048801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.049424 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.049426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.049493 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.050025 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.050037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.051967 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 09:00:34.052243 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.046432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53747 (local address 127.0.98.62:42923)
0620 09:00:34.046580 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.046586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.046605 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:00:34.046656 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.046661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.046731 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.046835 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.047195 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.048075 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.048914 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.049312 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.049443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.052128 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53747: BlockingRecv error: recv got EOF from 127.0.98.2:53747 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.058069 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.052917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:48499)
0620 09:00:34.053022 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.053040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.053165 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.053433 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.053436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.053592 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.053665 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.053669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.054406 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.054409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.054934 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.054941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.055395 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.055410 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.055485 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.055986 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.055997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.057912 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 09:00:34.058156 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.052997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48499 (local address 127.0.98.62:42923)
0620 09:00:34.053121 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.053125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.053139 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:34.053279 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.053282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.053326 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.053420 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.053776 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.054288 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.055053 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.055286 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.055396 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.058040 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48499: BlockingRecv error: recv got EOF from 127.0.98.2:48499 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 09:00:34.058372 9437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:00:34.154742 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:34.155069 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:34.155153 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:34.159447 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:34.159543 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:34.159636 9440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:34.165238 9440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35877
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:34.166383 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:34.167460 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:34.176920 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:34.177024 9445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:34.177089 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:34.177528 9440 server_base.cc:1034] running on GCE node
I20260620 09:00:34.177982 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:34.178635 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:34.179802 9440 hybrid_clock.cc:648] HybridClock initialized: now 1781946034179784 us; error 32 us; skew 500 ppm
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:34.182793 9440 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:00:34.183753 9440 webserver.cc:460] Webserver started at http://127.0.98.3:35877/ using document root <none> and password file <none>
I20260620 09:00:34.184365 9440 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:34.184509 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:34.188585 9440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 09:00:34.190701 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.191916 9440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:34.192062 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "3cc4a527e9be49739988fd5e1cf014c2"
format_stamp: "Formatted at 2026-06-20 09:00:33 on dist-test-slave-scsf"
server_key: "29e9aa91a4df27735ab14612c5ddcca0"
server_key_iv: "9e305c37def59de47990d355a1e5398f"
server_key_version: "encryptionkey@0"
I20260620 09:00:34.192675 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:34.203433 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:34.206857 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:34.207059 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:34.207679 9440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:34.208590 9440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:34.208655 9440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.208717 9440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:34.208779 9440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:34.218008 9440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43791
I20260620 09:00:34.218036 9568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43791 every 8 connection(s)
I20260620 09:00:34.219094 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:34.225896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9440
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:00:34.231065 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.221008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55401 (local address 127.0.98.62:42923)
0620 09:00:34.221147 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.221151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.222018 (+ 867us) server_negotiation.cc:407] Connection header received
0620 09:00:34.223479 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.223482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.223530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.223605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.224861 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.225377 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.226966 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.227305 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.227434 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.230957 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55401: BlockingRecv error: recv got EOF from 127.0.98.3:55401 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":55}
I20260620 09:00:34.231776 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.221435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:55401)
0620 09:00:34.221862 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.221903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.223241 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.223781 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.223787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.224140 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.224708 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.224724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.225487 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.225490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.226827 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.226840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.227469 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.227474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.227693 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.228340 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.228367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.230846 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
W20260620 09:00:34.232262 9569 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42923 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42923: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.237498 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.226944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.62:42923)
0620 09:00:34.227194 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.227198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.227211 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:34.227244 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.227247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.227292 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.227388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.228345 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.228931 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.229873 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.231348 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.231456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.233151 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:34.233161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:34.233181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:34.233213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:34.235025 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:34.235538 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:34.235547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:34.235557 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:34.235744 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:34.236050 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:34.236053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:34.236055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:34.236604 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:34.236777 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:34.237136 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 09:00:34.237371 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.238135 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.232824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:56017)
0620 09:00:34.233001 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.233017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.233089 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.233332 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.233334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.233459 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.233543 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.233547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.234270 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.234273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.234814 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.234822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.235335 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.235337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.235407 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.235844 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.235859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.238021 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 09:00:34.238276 9306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.232896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56017 (local address 127.0.98.62:42923)
0620 09:00:34.233009 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.233012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.233047 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:00:34.233182 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.233185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.233224 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.233313 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.233657 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.234142 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.234941 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.235213 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.235334 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.238132 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56017: BlockingRecv error: recv got EOF from 127.0.98.3:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.244024 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.238630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:59815)
0620 09:00:34.238941 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.238954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.239046 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.239267 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.239269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.239418 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.239503 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.239508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.240658 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.240661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.241243 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.241251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.241646 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.241649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.241722 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.242243 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.242255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.243896 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
W20260620 09:00:34.244125 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.238691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59815 (local address 127.0.98.62:42923)
0620 09:00:34.238829 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.238833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.239033 (+ 200us) server_negotiation.cc:407] Connection header received
0620 09:00:34.239123 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.239125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.239167 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.239249 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.239922 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.240569 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.241340 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.241545 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.241619 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.244010 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59815: BlockingRecv error: recv got EOF from 127.0.98.3:59815 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 09:00:34.244237 9569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:34 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:34.877969 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:34.872086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:53775)
0620 09:00:34.872516 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:34.872532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:34.872639 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:34.872902 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:34.872905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:34.873110 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:34.873220 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.873226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.873937 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.873939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.874517 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:34.874524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.874918 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.874921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:34.874978 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.875520 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:34.875530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:34.877794 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":184,"threads_started":1}
W20260620 09:00:34.878041 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:34.872211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53775 (local address 127.0.98.62:42923)
0620 09:00:34.872547 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:34.872551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:34.872592 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:00:34.872736 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:34.872739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:34.872783 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:34.872863 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:34.873321 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.873847 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.874616 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:34.874821 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:34.874894 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:34.877910 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53775: BlockingRecv error: recv got EOF from 127.0.98.1:53775 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:35.064844 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:35.058956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:56527)
0620 09:00:35.059324 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:35.059345 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:35.059481 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:35.059761 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:35.059764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:35.059981 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:35.060082 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.060105 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.060932 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.060937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.061519 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.061527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.061971 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.061974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.062057 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.062555 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:35.062566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:35.064663 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":183,"threads_started":1}
W20260620 09:00:35.064889 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:35.059066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56527 (local address 127.0.98.62:42923)
0620 09:00:35.059253 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:35.059258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:35.059435 (+ 177us) server_negotiation.cc:407] Connection header received
0620 09:00:35.059580 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:35.059584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:35.059635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:35.059740 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:35.060234 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.060821 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.061638 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.061857 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.061965 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.064784 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56527: BlockingRecv error: recv got EOF from 127.0.98.2:56527 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:35.249899 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:35.244915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:52279)
0620 09:00:35.245182 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:35.245195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:35.245290 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:35.245511 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:35.245513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:35.245685 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:35.245775 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.245780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.246468 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.246470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.247009 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.247016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.247417 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.247420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.247490 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.247941 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:35.247952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:35.249766 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
W20260620 09:00:35.250057 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:35.244971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52279 (local address 127.0.98.62:42923)
0620 09:00:35.245110 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:35.245114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:35.245259 (+ 145us) server_negotiation.cc:407] Connection header received
0620 09:00:35.245366 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:35.245368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:35.245410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:35.245480 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:35.245870 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.246377 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.247110 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.247321 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.247398 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.249902 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52279: BlockingRecv error: recv got EOF from 127.0.98.3:52279 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:35 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:35.884486 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:35.878840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:39627)
0620 09:00:35.879170 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:35.879185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:35.879277 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:35.879537 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:35.879539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:35.879718 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:35.879811 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.879816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.880695 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.880698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.881278 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:35.881286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.881740 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.881742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:35.881803 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.882288 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:35.882299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:35.884275 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
W20260620 09:00:35.884706 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:35.878969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39627 (local address 127.0.98.62:42923)
0620 09:00:35.879231 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:35.879234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:35.879250 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:35.879376 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:35.879378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:35.879421 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:35.879516 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:35.879913 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.880594 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.881394 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:35.881639 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:35.881726 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:35.884405 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39627: BlockingRecv error: recv got EOF from 127.0.98.1:39627 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":59,"threads_started":1}
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:36.071517 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:36.065857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:37489)
0620 09:00:36.066172 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:36.066191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:36.066331 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:36.066612 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:36.066615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:36.066820 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:36.066911 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.066917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.067687 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.067693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.068260 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.068268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.068828 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.068831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.068919 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.069462 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:36.069473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:36.071307 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
W20260620 09:00:36.071542 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:36.065931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37489 (local address 127.0.98.62:42923)
0620 09:00:36.066126 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:36.066132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:36.066271 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:00:36.066433 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:36.066437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:36.066494 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:36.066586 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:36.067043 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.067567 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.068392 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.068699 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.068824 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.071437 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37489: BlockingRecv error: recv got EOF from 127.0.98.2:37489 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:36.255806 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:36.250711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:49707)
0620 09:00:36.250996 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:36.251010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:36.251108 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:36.251324 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:36.251326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:36.251497 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:36.251598 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.251603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.252302 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.252304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.252942 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.252951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.253389 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.253391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.253450 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.253938 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:36.253949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:36.255679 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
W20260620 09:00:36.255990 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:36.250845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49707 (local address 127.0.98.62:42923)
0620 09:00:36.250996 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:36.251000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:36.251075 (+ 75us) server_negotiation.cc:407] Connection header received
0620 09:00:36.251178 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:36.251180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:36.251222 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:36.251289 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:36.251693 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.252212 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.253081 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.253295 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.253371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.255823 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49707: BlockingRecv error: recv got EOF from 127.0.98.3:49707 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:36 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:36.890480 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:36.885231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:59485)
0620 09:00:36.885516 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:36.885537 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:36.885620 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:36.885832 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:36.885834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:36.886001 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:36.886094 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.886099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.886794 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.886796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.887349 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:36.887356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.887791 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.887794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:36.887851 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.888357 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:36.888368 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:36.890344 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
W20260620 09:00:36.890647 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:36.885339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59485 (local address 127.0.98.62:42923)
0620 09:00:36.885621 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:36.885625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:36.885639 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:36.885677 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:36.885679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:36.885722 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:36.885801 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:36.886194 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.886694 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.887452 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:36.887677 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:36.887753 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:36.890477 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59485: BlockingRecv error: recv got EOF from 127.0.98.1:59485 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":154,"threads_started":1}
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:37.077896 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:37.072323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:37447)
0620 09:00:37.072649 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:37.072662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:37.072802 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:37.073137 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:37.073140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:37.073355 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:37.073458 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.073464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.074228 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.074234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.074801 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.074808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.075262 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.075265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.075346 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.075839 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:37.075850 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:37.077762 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
W20260620 09:00:37.078166 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:37.072449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37447 (local address 127.0.98.62:42923)
0620 09:00:37.072618 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:37.072623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:37.072749 (+ 126us) server_negotiation.cc:407] Connection header received
0620 09:00:37.072944 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:37.072946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:37.072989 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:37.073113 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:37.073585 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.074115 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.074929 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.075149 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.075264 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.077908 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37447: BlockingRecv error: recv got EOF from 127.0.98.2:37447 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:37.262735 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:37.256797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:53139)
0620 09:00:37.257098 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:37.257111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:37.257216 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:37.257481 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:37.257483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:37.257675 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:37.257779 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.257784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.258755 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.258761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.259556 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.259568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.260030 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.260032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.260091 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.260667 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:37.260678 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:37.262616 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":116,"threads_started":1}
W20260620 09:00:37.262903 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:37.256871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53139 (local address 127.0.98.62:42923)
0620 09:00:37.257041 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:37.257047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:37.257187 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:00:37.257307 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:37.257310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:37.257358 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:37.257446 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:37.257981 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.258606 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.259695 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.259918 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.260022 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.262748 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53139: BlockingRecv error: recv got EOF from 127.0.98.3:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:37 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:37.900040 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:37.891518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:53293)
0620 09:00:37.891778 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:37.891793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:37.891923 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:37.892216 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:37.892218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:37.892484 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:37.892600 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.892606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.893526 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.893528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.894076 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:37.894084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.894559 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.894561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:37.894618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.898049 (+ 3431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:37.898062 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:37.899889 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260620 09:00:37.900202 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:37.891649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53293 (local address 127.0.98.62:42923)
0620 09:00:37.891902 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:37.891906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:37.891920 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:37.892029 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:37.892031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:37.892085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:37.892179 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:37.892713 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.893428 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.894170 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:37.894469 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:37.894542 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:37.900066 (+ 5524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53293: BlockingRecv error: recv got EOF from 127.0.98.1:53293 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:38.084308 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:38.078829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:47571)
0620 09:00:38.079125 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:38.079138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:38.079269 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:38.079582 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:38.079585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:38.079841 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:38.079996 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.080023 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.080900 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.080903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.081467 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.081475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.081859 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.081862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.081919 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.082407 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:38.082417 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:38.084192 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
W20260620 09:00:38.084483 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:38.078922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47571 (local address 127.0.98.62:42923)
0620 09:00:38.079079 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:38.079083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:38.079225 (+ 142us) server_negotiation.cc:407] Connection header received
0620 09:00:38.079398 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:38.079401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:38.079455 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:38.079540 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:38.080162 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.080801 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.081570 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.081769 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.081841 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.084332 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47571: BlockingRecv error: recv got EOF from 127.0.98.2:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:38.269368 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:38.263631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:54559)
0620 09:00:38.263907 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:38.263921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:38.264022 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:38.264253 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:38.264255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:38.264438 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:38.264559 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.264565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.265319 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.265321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.265895 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.265902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.266336 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.266342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.266421 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.267022 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:38.267032 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:38.269186 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
W20260620 09:00:38.269423 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:38.263759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54559 (local address 127.0.98.62:42923)
0620 09:00:38.263899 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:38.263903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:38.263991 (+ 88us) server_negotiation.cc:407] Connection header received
0620 09:00:38.264091 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:38.264093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:38.264140 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:38.264217 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:38.264655 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.265224 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.265998 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.266219 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.266296 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.269313 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54559: BlockingRecv error: recv got EOF from 127.0.98.3:54559 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:38 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:38.906167 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:38.900941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:49551)
0620 09:00:38.901210 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:38.901224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:38.901306 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:38.901542 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:38.901544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:38.901714 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:38.901807 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.901812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.902527 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.902529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.903091 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:38.903098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.903518 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.903521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:38.903579 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.904071 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:38.904082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:38.906048 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20260620 09:00:38.906329 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:38.900989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49551 (local address 127.0.98.62:42923)
0620 09:00:38.901245 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:38.901248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:38.901264 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:38.901390 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:38.901392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:38.901434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:38.901503 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:38.901910 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.902435 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.903201 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:38.903422 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:38.903499 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:38.906193 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49551: BlockingRecv error: recv got EOF from 127.0.98.1:49551 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:39.091053 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:39.085245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:45495)
0620 09:00:39.085596 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:39.085612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:39.085747 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:39.086054 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:39.086056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:39.086260 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:39.086346 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.086350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.087173 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.087179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.087821 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.087830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.088303 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.088307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.088401 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.088992 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:39.089003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:39.090865 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
W20260620 09:00:39.091130 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:39.085451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45495 (local address 127.0.98.62:42923)
0620 09:00:39.085597 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:39.085603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:39.085697 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:00:39.085858 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:39.085861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:39.085928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:39.086033 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:39.086474 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.087064 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.087957 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.088184 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.088298 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.091000 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45495: BlockingRecv error: recv got EOF from 127.0.98.2:45495 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:39.276088 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:39.270290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:54207)
0620 09:00:39.270615 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:39.270628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:39.270741 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:39.270967 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:39.270969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:39.271138 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:39.271226 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.271230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.271963 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.271966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.272575 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.272588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.273061 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.273064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.273141 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.273753 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:39.273764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:39.275903 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":85,"threads_started":1}
W20260620 09:00:39.276237 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:39.270463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54207 (local address 127.0.98.62:42923)
0620 09:00:39.270625 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:39.270628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:39.270697 (+ 69us) server_negotiation.cc:407] Connection header received
0620 09:00:39.270821 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:39.270823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:39.270868 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:39.270935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:39.271324 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.271830 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.272695 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.272950 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.273031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.276079 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54207: BlockingRecv error: recv got EOF from 127.0.98.3:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:39 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:39.912577 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:39.906945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:51993)
0620 09:00:39.907241 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:39.907257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:39.907365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:39.907658 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:39.907661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:39.907869 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:39.907961 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.907966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.908870 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.908875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.909437 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:39.909444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.909936 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.909940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:39.910025 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.910514 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:39.910525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:39.912396 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
W20260620 09:00:39.912770 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:39.907005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51993 (local address 127.0.98.62:42923)
0620 09:00:39.907313 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:39.907317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:39.907331 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:39.907481 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:39.907485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:39.907539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:39.907632 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:39.908085 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.908767 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.909583 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:39.909825 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:39.909931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:39.912596 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51993: BlockingRecv error: recv got EOF from 127.0.98.1:51993 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:40.098018 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:40.091953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:58267)
0620 09:00:40.092248 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:40.092262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:40.092419 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:40.092717 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:40.092719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:40.092908 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:40.093005 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.093010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.093955 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.093959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.094856 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.094866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.095440 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.095443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.095506 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.095995 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:40.096007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:40.097902 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
W20260620 09:00:40.098160 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:40.092028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58267 (local address 127.0.98.62:42923)
0620 09:00:40.092184 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:40.092188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:40.092364 (+ 176us) server_negotiation.cc:407] Connection header received
0620 09:00:40.092535 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:40.092538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:40.092604 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:40.092680 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:40.093132 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.093855 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.095038 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.095337 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.095421 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.098043 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58267: BlockingRecv error: recv got EOF from 127.0.98.2:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:40.282577 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:40.277028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:51195)
0620 09:00:40.277352 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:40.277365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:40.277506 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:40.277811 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:40.277814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:40.278005 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:40.278095 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.278100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.278869 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.278874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.279429 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.279437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.279912 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.279916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.280001 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.280603 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:40.280616 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:40.282405 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
W20260620 09:00:40.282779 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:40.277108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51195 (local address 127.0.98.62:42923)
0620 09:00:40.277282 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:40.277288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:40.277442 (+ 154us) server_negotiation.cc:407] Connection header received
0620 09:00:40.277608 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:40.277611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:40.277667 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:40.277757 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:40.278220 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.278755 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.279550 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.279792 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.279908 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.282561 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51195: BlockingRecv error: recv got EOF from 127.0.98.3:51195 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:40 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:40.919130 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:40.913402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:50913)
0620 09:00:40.913783 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:40.913797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:40.913896 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:40.914140 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:40.914142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:40.914321 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:40.914417 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.914422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.915262 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.915271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.915917 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:40.915927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.916382 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.916391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:40.916481 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.917125 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:40.917136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:40.918980 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
W20260620 09:00:40.919338 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:40.913469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50913 (local address 127.0.98.62:42923)
0620 09:00:40.913734 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:40.913737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:40.913864 (+ 127us) server_negotiation.cc:407] Connection header received
0620 09:00:40.913970 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:40.913972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:40.914022 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:40.914102 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:40.914527 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.915087 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.916032 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:40.916266 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:40.916348 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:40.919119 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50913: BlockingRecv error: recv got EOF from 127.0.98.1:50913 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:41.104837 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:41.098852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:35721)
0620 09:00:41.099191 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:41.099204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:41.099337 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:41.099600 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:41.099603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:41.099825 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:41.099915 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.099920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.100745 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.100751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.101337 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.101344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.101798 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.101801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.101886 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.102566 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:41.102583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:41.104717 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
W20260620 09:00:41.104998 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:41.098926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35721 (local address 127.0.98.62:42923)
0620 09:00:41.099109 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:41.099114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:41.099278 (+ 164us) server_negotiation.cc:407] Connection header received
0620 09:00:41.099436 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:41.099440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:41.099493 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:41.099582 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:41.100030 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.100617 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.101459 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.101689 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.101794 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.104858 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35721: BlockingRecv error: recv got EOF from 127.0.98.2:35721 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:41.289037 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:41.283436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:47659)
0620 09:00:41.283787 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:41.283803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:41.283951 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:41.284240 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:41.284243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:41.284478 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:41.284566 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.284571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.285334 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.285339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.285911 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.285919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.286363 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.286366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.286448 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.286963 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:41.286974 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:41.288918 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
W20260620 09:00:41.289199 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:41.283571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47659 (local address 127.0.98.62:42923)
0620 09:00:41.283723 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:41.283729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:41.283898 (+ 169us) server_negotiation.cc:407] Connection header received
0620 09:00:41.284073 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:41.284080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:41.284127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:41.284215 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:41.284694 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.285220 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.286029 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.286251 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.286359 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.289063 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47659: BlockingRecv error: recv got EOF from 127.0.98.3:47659 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:41 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:41.925905 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:41.920056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:34483)
0620 09:00:41.920338 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:41.920351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:41.920440 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:41.920740 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:41.920743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:41.920929 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:41.921020 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.921024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.921795 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.921801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.922351 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:41.922358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.922844 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.922847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:41.922913 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.923423 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:41.923435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:41.925666 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
W20260620 09:00:41.926043 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:41.920184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34483 (local address 127.0.98.62:42923)
0620 09:00:41.920402 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:41.920405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:41.920419 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:41.920554 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:41.920556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:41.920624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:41.920704 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:41.921134 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.921678 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.922476 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:41.922720 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:41.922809 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:41.925864 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34483: BlockingRecv error: recv got EOF from 127.0.98.1:34483 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":57,"threads_started":1}
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:42.111377 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:42.105637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:42245)
0620 09:00:42.105910 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:42.105923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:42.106019 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:42.106241 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:42.106243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:42.106633 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:42.106730 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.106736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.107679 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.107682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.108212 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.108219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.108788 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.108791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.108854 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.109439 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:42.109450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:42.111263 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
W20260620 09:00:42.111558 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:42.105779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42245 (local address 127.0.98.62:42923)
0620 09:00:42.105922 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:42.105926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:42.105987 (+ 61us) server_negotiation.cc:407] Connection header received
0620 09:00:42.106096 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:42.106098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:42.106141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:42.106208 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:42.106850 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.107557 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.108368 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.108690 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.108773 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.111423 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42245: BlockingRecv error: recv got EOF from 127.0.98.2:42245 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:42.295811 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:42.289977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:33411)
0620 09:00:42.290262 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:42.290276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:42.290371 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:42.290611 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:42.290613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:42.290804 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:42.290895 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.290900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.291607 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.291609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.292335 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.292346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.292904 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.292907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.292995 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.293552 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:42.293566 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:42.295643 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260620 09:00:42.295840 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:42.290082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33411 (local address 127.0.98.62:42923)
0620 09:00:42.290222 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:42.290225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:42.290343 (+ 118us) server_negotiation.cc:407] Connection header received
0620 09:00:42.290450 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:42.290452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:42.290499 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:42.290574 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:42.290997 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.291515 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.292483 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.292788 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.292891 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.295742 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33411: BlockingRecv error: recv got EOF from 127.0.98.3:33411 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:42 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:42.936177 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:42.927064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:38855)
0620 09:00:42.927356 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:42.927376 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:42.927498 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:42.927852 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:42.927858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:42.928129 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:42.928267 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.928289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.929353 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.929356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.930024 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:42.930033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.930540 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.930543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:42.930605 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.934131 (+ 3526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:42.934143 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:42.936055 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
W20260620 09:00:42.936508 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:42.927097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38855 (local address 127.0.98.62:42923)
0620 09:00:42.927415 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:42.927419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:42.927435 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:42.927619 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:42.927622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:42.927693 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:42.927790 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:42.928423 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.929256 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.930181 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:42.930434 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:42.930518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:42.936214 (+ 5696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38855: BlockingRecv error: recv got EOF from 127.0.98.1:38855 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":74,"threads_started":1}
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:43.118494 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:43.112273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:46745)
0620 09:00:43.112694 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:43.112707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:43.112843 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:43.113233 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:43.113236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:43.113468 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:43.113592 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.113598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.114592 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.114595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.115162 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.115170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.115704 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.115707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.115812 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.116298 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:43.116309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:43.118354 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
W20260620 09:00:43.118659 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:43.112429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46745 (local address 127.0.98.62:42923)
0620 09:00:43.112617 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:43.112622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:43.112784 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:00:43.112990 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:43.112992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:43.113082 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:43.113176 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:43.113727 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.114462 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.115357 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.115602 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.115684 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.118511 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46745: BlockingRecv error: recv got EOF from 127.0.98.2:46745 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:43.302378 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:43.296597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:48511)
0620 09:00:43.296907 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:43.296920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:43.297048 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:43.297345 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:43.297348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:43.297549 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:43.297651 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.297656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.298400 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.298406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.298957 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.298965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.299408 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.299412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.299493 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.300029 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:43.300041 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:43.302211 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
W20260620 09:00:43.302423 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:43.296663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48511 (local address 127.0.98.62:42923)
0620 09:00:43.296830 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:43.296835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:43.297011 (+ 176us) server_negotiation.cc:407] Connection header received
0620 09:00:43.297174 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:43.297179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:43.297230 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:43.297321 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:43.297766 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.298280 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.299071 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.299296 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.299405 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.302325 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48511: BlockingRecv error: recv got EOF from 127.0.98.3:48511 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:43 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:43.942660 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:43.937074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:51177)
0620 09:00:43.937360 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:43.937374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:43.937465 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:43.937711 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:43.937713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:43.937887 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:43.937981 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.937986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.938781 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.938784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.939572 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:43.939592 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.940183 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.940187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:43.940278 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.940824 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:43.940835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:43.942538 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":105,"threads_started":1}
W20260620 09:00:43.942857 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:43.937154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51177 (local address 127.0.98.62:42923)
0620 09:00:43.937439 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:43.937443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:43.937457 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:00:43.937558 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:43.937561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:43.937603 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:43.937674 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:43.938102 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.938660 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.939699 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:43.940043 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:43.940149 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:43.942688 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51177: BlockingRecv error: recv got EOF from 127.0.98.1:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":67,"threads_started":1}
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:44.124912 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:44.119287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:59871)
0620 09:00:44.119776 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:44.119810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:44.119923 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:44.120163 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:44.120165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:44.120343 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:44.120434 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.120439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.121213 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.121215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.121775 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.121794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.122198 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.122200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.122257 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.122764 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:44.122775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:44.124737 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":118,"threads_started":1}
W20260620 09:00:44.124989 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:44.119356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59871 (local address 127.0.98.62:42923)
0620 09:00:44.119571 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:44.119578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:44.119891 (+ 313us) server_negotiation.cc:407] Connection header received
0620 09:00:44.120004 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:44.120007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:44.120055 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:44.120126 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:44.120598 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.121124 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.121889 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.122103 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.122179 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.124862 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59871: BlockingRecv error: recv got EOF from 127.0.98.2:59871 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:44.308856 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:44.303200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:46477)
0620 09:00:44.303526 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:44.303541 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:44.303635 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:44.303876 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:44.303878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:44.304060 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:44.304193 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.304201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.305259 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.305262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.305906 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.305916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.306326 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.306329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.306383 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.306903 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:44.306915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:44.308711 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":149,"threads_started":1}
W20260620 09:00:44.309032 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:44.303385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46477 (local address 127.0.98.62:42923)
0620 09:00:44.303521 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:44.303525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:44.303618 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:00:44.303714 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:44.303716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:44.303761 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:44.303840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:44.304346 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.305135 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.306007 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.306232 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.306310 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.308847 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46477: BlockingRecv error: recv got EOF from 127.0.98.3:46477 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:44 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:44.948406 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:44.943424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:56969)
0620 09:00:44.943680 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:44.943693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:44.943789 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:44.944010 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:44.944012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:44.944185 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:44.944280 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.944285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.945090 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.945093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.945649 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:44.945656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.946063 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.946065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:44.946120 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.946594 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:44.946605 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:44.948275 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
W20260620 09:00:44.948619 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:44.943473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56969 (local address 127.0.98.62:42923)
0620 09:00:44.943705 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:44.943708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:44.943742 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:00:44.943855 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:44.943857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:44.943898 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:44.943977 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:44.944380 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.944989 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.945761 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:44.945973 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:44.946048 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:44.948416 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56969: BlockingRecv error: recv got EOF from 127.0.98.1:56969 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:45.131487 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:45.125924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:56971)
0620 09:00:45.126216 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:45.126229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:45.126333 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:45.126579 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:45.126581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:45.126769 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:45.126861 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.126865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.127569 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.127571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.128137 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.128144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.128821 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.128825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.128938 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.129535 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:45.129547 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:45.131353 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
W20260620 09:00:45.131680 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:45.126042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56971 (local address 127.0.98.62:42923)
0620 09:00:45.126240 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:45.126244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:45.126302 (+ 58us) server_negotiation.cc:407] Connection header received
0620 09:00:45.126414 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:45.126416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:45.126467 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:45.126539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:45.126955 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.127446 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.128242 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.128685 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.128809 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.131522 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56971: BlockingRecv error: recv got EOF from 127.0.98.2:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:45.315884 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:45.309838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:54537)
0620 09:00:45.310158 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:45.310177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:45.310335 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:45.310720 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:45.310723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:45.311013 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:45.311172 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.311179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.311966 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.311970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.312552 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.312560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.313012 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.313016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.313100 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.313684 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:45.313701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:45.315708 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
W20260620 09:00:45.315968 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:45.309930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54537 (local address 127.0.98.62:42923)
0620 09:00:45.310074 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:45.310081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:45.310252 (+ 171us) server_negotiation.cc:407] Connection header received
0620 09:00:45.310536 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:45.310539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:45.310603 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:45.310694 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:45.311294 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.311844 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.312677 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.312897 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.313007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.315837 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54537: BlockingRecv error: recv got EOF from 127.0.98.3:54537 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:45 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:45.955464 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:45.949268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:41993)
0620 09:00:45.949621 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:45.949635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:45.949739 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:45.950055 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:45.950057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:45.950231 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:45.950326 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.950331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.951106 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.951108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.951837 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:45.951850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.952534 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.952538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:45.952627 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.953223 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:45.953235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:45.955347 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":152,"threads_started":1}
W20260620 09:00:45.955636 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:45.949337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41993 (local address 127.0.98.62:42923)
0620 09:00:45.949632 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:45.949636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:45.949708 (+ 72us) server_negotiation.cc:407] Connection header received
0620 09:00:45.949873 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:45.949876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:45.949933 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:45.950016 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:45.950441 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.951011 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.952003 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:45.952339 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:45.952487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:45.955499 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41993: BlockingRecv error: recv got EOF from 127.0.98.1:41993 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:46.137815 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:46.132369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:58381)
0620 09:00:46.132734 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:46.132751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:46.132886 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:46.133173 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:46.133176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:46.133392 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:46.133495 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.133501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.134271 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.134275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.134847 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.134854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.135308 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.135311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.135403 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.135884 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:46.135894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:46.137645 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
W20260620 09:00:46.137876 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:46.132538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58381 (local address 127.0.98.62:42923)
0620 09:00:46.132703 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:46.132708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:46.132837 (+ 129us) server_negotiation.cc:407] Connection header received
0620 09:00:46.133008 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:46.133011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:46.133064 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:46.133154 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:46.133623 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.134158 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.134975 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.135197 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.135303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.137772 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58381: BlockingRecv error: recv got EOF from 127.0.98.2:58381 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:46.322144 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:46.316823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:56549)
0620 09:00:46.317142 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:46.317156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:46.317251 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:46.317472 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:46.317474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:46.317655 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:46.317749 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.317754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.318558 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.318560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.319139 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.319147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.319608 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.319611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.319674 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.320141 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:46.320152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:46.322023 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":151,"threads_started":1}
W20260620 09:00:46.322295 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:46.316883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56549 (local address 127.0.98.62:42923)
0620 09:00:46.316996 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:46.317000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:46.317221 (+ 221us) server_negotiation.cc:407] Connection header received
0620 09:00:46.317317 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:46.317320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:46.317366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:46.317436 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:46.317846 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.318453 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.319240 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.319520 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.319602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.322175 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56549: BlockingRecv error: recv got EOF from 127.0.98.3:56549 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:46 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:46.962162 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:46.956560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:52593)
0620 09:00:46.956825 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:46.956838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:46.956939 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:46.957204 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:46.957206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:46.957378 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:46.957472 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.957477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.958207 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.958210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.958747 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:46.958755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.959162 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.959164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:46.959221 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.959817 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:46.959828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:46.962045 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
W20260620 09:00:46.962322 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:46.956638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52593 (local address 127.0.98.62:42923)
0620 09:00:46.956877 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:46.956880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:46.956896 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:46.957055 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:46.957057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:46.957105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:46.957189 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:46.957567 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.958111 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.958853 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:46.959070 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:46.959145 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:46.962183 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52593: BlockingRecv error: recv got EOF from 127.0.98.1:52593 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:47.143822 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:47.138748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:43499)
0620 09:00:47.139012 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:47.139031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:47.139122 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:47.139340 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:47.139342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:47.139528 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:47.139620 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.139625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.140320 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.140322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.140955 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.140964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.141386 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.141388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.141458 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.141956 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:47.141967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:47.143703 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
W20260620 09:00:47.143965 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:47.138873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43499 (local address 127.0.98.62:42923)
0620 09:00:47.139011 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:47.139015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:47.139094 (+ 79us) server_negotiation.cc:407] Connection header received
0620 09:00:47.139186 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:47.139188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:47.139230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:47.139309 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:47.139716 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.140234 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.141062 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.141299 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.141388 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.143842 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43499: BlockingRecv error: recv got EOF from 127.0.98.2:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:47.328581 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:47.323060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:54779)
0620 09:00:47.323361 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:47.323379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:47.323488 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:47.323804 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:47.323810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:47.324044 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:47.324175 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.324182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.325081 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.325083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.325632 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.325640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.326034 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.326036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.326091 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.326546 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:47.326556 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:47.328390 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260620 09:00:47.328663 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:47.323113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54779 (local address 127.0.98.62:42923)
0620 09:00:47.323268 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:47.323275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:47.323463 (+ 188us) server_negotiation.cc:407] Connection header received
0620 09:00:47.323597 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:47.323599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:47.323664 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:47.323754 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:47.324301 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.324969 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.325727 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.325945 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.326018 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.328542 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54779: BlockingRecv error: recv got EOF from 127.0.98.3:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:47 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:47.974440 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:47.962955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:43541)
0620 09:00:47.963245 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:47.963264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:47.963395 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:47.963634 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:47.963636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:47.963812 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:47.963908 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.963913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.964750 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.964754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.965593 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:47.965602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.966083 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.966086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:47.966144 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.972243 (+ 6099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:47.972257 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:47.974294 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
W20260620 09:00:47.974671 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:47.963081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43541 (local address 127.0.98.62:42923)
0620 09:00:47.963420 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:47.963424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:47.963439 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:00:47.963479 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:47.963481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:47.963523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:47.963603 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:47.964010 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.964636 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.965728 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:47.965987 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:47.966069 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:47.974468 (+ 8399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43541: BlockingRecv error: recv got EOF from 127.0.98.1:43541 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":79,"threads_started":1}
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:48.149869 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:48.144718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:32965)
0620 09:00:48.145016 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:48.145029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:48.145121 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:48.145360 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:48.145362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:48.145538 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:48.145631 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.145636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.146333 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.146335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.146867 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.146875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.147282 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.147285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.147342 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.147800 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:48.147810 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:48.149730 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
W20260620 09:00:48.150053 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:48.144915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32965 (local address 127.0.98.62:42923)
0620 09:00:48.145046 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:48.145049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:48.145077 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:00:48.145208 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:48.145210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:48.145253 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:48.145323 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:48.145737 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.146230 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.146960 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.147161 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.147234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.149917 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32965: BlockingRecv error: recv got EOF from 127.0.98.2:32965 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:48.334515 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:48.329335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:55105)
0620 09:00:48.329603 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:48.329617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:48.329712 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:48.329964 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:48.329966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:48.330137 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:48.330228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.330233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.330957 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.330960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.331542 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.331549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.331967 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.331970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.332026 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.332575 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:48.332587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:48.334395 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260620 09:00:48.334820 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:48.329439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55105 (local address 127.0.98.62:42923)
0620 09:00:48.329607 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:48.329610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:48.329668 (+ 58us) server_negotiation.cc:407] Connection header received
0620 09:00:48.329793 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:48.329795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:48.329861 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:48.329932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:48.330350 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.330862 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.331655 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.331873 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.331952 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.334557 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55105: BlockingRecv error: recv got EOF from 127.0.98.3:55105 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:48 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:48.980868 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:48.975402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:43197)
0620 09:00:48.975665 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:48.975679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:48.975763 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:48.976011 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:48.976013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:48.976184 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:48.976295 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.976301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.977038 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.977041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.977650 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:48.977662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.978258 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.978262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:48.978373 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.978950 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:48.978962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:48.980756 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260620 09:00:48.981072 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:48.975514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43197 (local address 127.0.98.62:42923)
0620 09:00:48.975753 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:48.975756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:48.975772 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:00:48.975864 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:48.975866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:48.975907 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:48.975980 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:48.976399 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.976954 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.977772 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:48.978142 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:48.978253 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:48.980896 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43197: BlockingRecv error: recv got EOF from 127.0.98.1:43197 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
Jun 20 09:00:49 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:00:49 dist-test-slave-scsf krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:00:49.156555 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.150889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:37051)
0620 09:00:49.151166 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:49.151180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:49.151275 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:49.151509 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:49.151511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:49.151684 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:49.151784 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.151789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.152552 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.152555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.153217 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.153228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.153667 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.153669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.153739 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.154223 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:49.154238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:49.156367 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42923: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
W20260620 09:00:49.156709 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:49.151010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37051 (local address 127.0.98.62:42923)
0620 09:00:49.151147 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:49.151150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:49.151256 (+ 106us) server_negotiation.cc:407] Connection header received
0620 09:00:49.151359 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:49.151361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:49.151403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:49.151473 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:49.151877 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.152378 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.153328 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.153561 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.153642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.156589 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37051: BlockingRecv error: recv got EOF from 127.0.98.2:37051 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:00:49.235505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9156
I20260620 09:00:49.241250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9308
I20260620 09:00:49.246537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9440
I20260620 09:00:49.251781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9086
2026-06-20T09:00:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18015 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T09:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:49Z Disabled control of system clock
I20260620 09:00:49.282280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44245
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:00:49.393430 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:49.393707 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:49.393741 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:49.397707 9637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:49.397769 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:49.397784 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:49.397801 9637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:49.397822 9637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:49.402539 9637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:49.403595 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:49.404934 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:00:49.413136 9637 server_base.cc:1034] running on GCE node
W20260620 09:00:49.413110 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.413095 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.413353 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:49.413954 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:49.415027 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:49.416193 9637 hybrid_clock.cc:648] HybridClock initialized: now 1781946049416178 us; error 26 us; skew 500 ppm
I20260620 09:00:49.418138 9637 webserver.cc:460] Webserver started at http://127.0.98.62:38709/ using document root <none> and password file <none>
I20260620 09:00:49.418687 9637 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:49.418752 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:49.418918 9637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:49.420984 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "2dba821797ded96b1c6a80eb3d7baa78"
server_key_iv: "a22df2f6f31bd70e235e77c1e589beb0"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.421576 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "2dba821797ded96b1c6a80eb3d7baa78"
server_key_iv: "a22df2f6f31bd70e235e77c1e589beb0"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.425567 9637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:49.427913 9651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.429147 9637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:49.429313 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "2dba821797ded96b1c6a80eb3d7baa78"
server_key_iv: "a22df2f6f31bd70e235e77c1e589beb0"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.429430 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:49.446499 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:49.447261 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:49.447460 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:49.454818 9637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44245
I20260620 09:00:49.454833 9703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44245 every 8 connection(s)
I20260620 09:00:49.455807 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:49.457957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9637
I20260620 09:00:49.458077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:49.458398 392 external_mini_cluster.cc:1428] Setting key 0790a83dbdf4f3413640aac117518052
I20260620 09:00:49.459411 9704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:49.465250 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: Bootstrap starting.
I20260620 09:00:49.467630 9704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:49.467839 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.459872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.98.62:44245)
0620 09:00:49.460611 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:49.460635 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:49.460688 (+ 53us) server_negotiation.cc:407] Connection header received
0620 09:00:49.461380 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:49.461397 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:49.461800 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:49.462132 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:49.462569 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.463543 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.464422 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.464955 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.465126 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.465988 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:49.465993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:49.466006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:49.466432 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:49.466490 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:49.466510 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:49.466611 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:49.466821 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 09:00:49.467069 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":137,"threads_started":1}
I20260620 09:00:49.468593 9704 log.cc:824] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:49.470566 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: No bootstrap required, opened a new log
I20260620 09:00:49.474129 9704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } }
I20260620 09:00:49.474355 9704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:49.474390 9704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5f79f7bfb3468b9fb573e5d6e9ffe9, State: Initialized, Role: FOLLOWER
I20260620 09:00:49.474748 9704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } }
I20260620 09:00:49.474897 9704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:49.474977 9704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:49.475104 9704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:49.476019 9704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } }
I20260620 09:00:49.476372 9704 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5f79f7bfb3468b9fb573e5d6e9ffe9; no voters:
I20260620 09:00:49.476702 9704 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:49.476852 9709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:49.477087 9709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f5f79f7bfb3468b9fb573e5d6e9ffe9, State: Running, Role: LEADER
I20260620 09:00:49.477460 9709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } }
I20260620 09:00:49.477967 9704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:49.479038 9710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } } }
I20260620 09:00:49.479117 9711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5f79f7bfb3468b9fb573e5d6e9ffe9. Latest consensus state: current_term: 1 leader_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f79f7bfb3468b9fb573e5d6e9ffe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44245 } } }
I20260620 09:00:49.479176 9711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:49.479176 9710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9 [sys.catalog]: This master's current role is: LEADER
W20260620 09:00:49.482690 9724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:49.482784 9724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:49.482867 9718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:49.483546 9718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:49.489207 9718 catalog_manager.cc:1347] Generated new cluster ID: 22761f3defc34f709f6a25b40a0c3522
I20260620 09:00:49.489288 9718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:49.523494 9718 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:49.524559 9718 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:49.541322 9718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5f79f7bfb3468b9fb573e5d6e9ffe9: Generated new TSK 0
I20260620 09:00:49.542136 9718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:49.553100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--builtin_ntp_servers=127.0.98.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 09:00:49.669615 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:49.669903 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:49.669941 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:49.674165 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:49.674229 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:49.674329 9728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:49.679503 9728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:49.680702 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:49.682108 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:49.691715 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.691723 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.691723 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:49.691974 9728 server_base.cc:1034] running on GCE node
I20260620 09:00:49.692531 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:49.693203 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:49.694391 9728 hybrid_clock.cc:648] HybridClock initialized: now 1781946049694360 us; error 50 us; skew 500 ppm
I20260620 09:00:49.696147 9728 webserver.cc:460] Webserver started at http://127.0.98.1:43493/ using document root <none> and password file <none>
I20260620 09:00:49.696797 9728 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:49.696885 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:49.697106 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:49.699304 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "088aa3317a6f4dc29ae5fdefaff8eff5"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "44c3a3fef724873dae41196f718078c1"
server_key_iv: "295faa201317bd6d15e8b1ef1de91674"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.699952 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "088aa3317a6f4dc29ae5fdefaff8eff5"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "44c3a3fef724873dae41196f718078c1"
server_key_iv: "295faa201317bd6d15e8b1ef1de91674"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.703553 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.003s
I20260620 09:00:49.706104 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.707306 9728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:49.707441 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "088aa3317a6f4dc29ae5fdefaff8eff5"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "44c3a3fef724873dae41196f718078c1"
server_key_iv: "295faa201317bd6d15e8b1ef1de91674"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.707556 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:49.727507 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:49.728288 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:49.728550 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:49.729137 9728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:49.730067 9728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:49.730134 9728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.730201 9728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:49.730247 9728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.739374 9728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43151
I20260620 09:00:49.739392 9855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43151 every 8 connection(s)
I20260620 09:00:49.740302 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:49.749480 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.742116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59657 (local address 127.0.98.62:44245)
0620 09:00:49.742250 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:49.742254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:49.743266 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 09:00:49.744721 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:49.744725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:49.744788 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:49.744890 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:49.745786 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.746325 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.747460 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.747690 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.747801 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.748955 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:49.748959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:49.748966 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:49.748978 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:49.748996 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:49.749004 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:49.749082 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:49.749229 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 09:00:49.749327 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:00:49.749559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9728
I20260620 09:00:49.749679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:49.749955 392 external_mini_cluster.cc:1428] Setting key 6ee989d4dd0ead17846b33455baa52eb
I20260620 09:00:49.750491 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.742458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44245 (local address 127.0.98.1:59657)
0620 09:00:49.743123 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:49.743165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:49.744513 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:49.745065 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:49.745072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:49.745159 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:49.745646 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.745657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.746447 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.746452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.747337 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.747346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.747848 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.747852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.748016 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.748663 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:49.748664 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:49.748706 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:49.748795 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:49.749083 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:49.749094 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:49.749102 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:00:49.749229 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:00:49.749508 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260620 09:00:49.751672 9856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44245
I20260620 09:00:49.751860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--builtin_ntp_servers=127.0.98.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:00:49.751904 9856 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:49.752439 9856 heartbeater.cc:511] Master 127.0.98.62:44245 requested a full tablet report, sending...
I20260620 09:00:49.753952 9667 ts_manager.cc:194] Registered new tserver with Master: 088aa3317a6f4dc29ae5fdefaff8eff5 (127.0.98.1:43151)
I20260620 09:00:49.755085 9667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59657
W20260620 09:00:49.862823 9859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:49.863108 9859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:49.863144 9859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:49.867266 9859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:49.867333 9859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:49.867413 9859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:49.871939 9859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:49.873051 9859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:49.874418 9859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:49.884199 9867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.884089 9865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:49.884089 9864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:49.884958 9859 server_base.cc:1034] running on GCE node
I20260620 09:00:49.885339 9859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:49.885924 9859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:49.887084 9859 hybrid_clock.cc:648] HybridClock initialized: now 1781946049887054 us; error 43 us; skew 500 ppm
I20260620 09:00:49.888952 9859 webserver.cc:460] Webserver started at http://127.0.98.2:40803/ using document root <none> and password file <none>
I20260620 09:00:49.889549 9859 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:49.889611 9859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:49.889864 9859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:49.891878 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6c45c1587f941ec94ad48ce76e6a88a"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "92aaf52b39ad21a9dc5820fe2c8b6ab4"
server_key_iv: "7de176fb34fb6496a18f4070cfc7bd46"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.892429 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6c45c1587f941ec94ad48ce76e6a88a"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "92aaf52b39ad21a9dc5820fe2c8b6ab4"
server_key_iv: "7de176fb34fb6496a18f4070cfc7bd46"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.896242 9859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:00:49.898775 9873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.900023 9859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:49.900141 9859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "b6c45c1587f941ec94ad48ce76e6a88a"
format_stamp: "Formatted at 2026-06-20 09:00:49 on dist-test-slave-scsf"
server_key: "92aaf52b39ad21a9dc5820fe2c8b6ab4"
server_key_iv: "7de176fb34fb6496a18f4070cfc7bd46"
server_key_version: "encryptionkey@0"
I20260620 09:00:49.900249 9859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:49.909731 9859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:49.910444 9859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:49.910616 9859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:49.911149 9859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:49.912089 9859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:49.912134 9859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.912209 9859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:49.912243 9859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:49.921164 9859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37363
I20260620 09:00:49.921178 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37363 every 8 connection(s)
I20260620 09:00:49.922075 9859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:49.927886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9859
I20260620 09:00:49.927984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:49.928270 392 external_mini_cluster.cc:1428] Setting key b880df0113870b83f6720ad406a1409e
I20260620 09:00:49.930441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--builtin_ntp_servers=127.0.98.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:00:49.931515 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.923827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39401 (local address 127.0.98.62:44245)
0620 09:00:49.923960 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:49.923965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:49.925086 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 09:00:49.926474 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:49.926478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:49.926539 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:49.926665 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:49.927579 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.928191 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.929530 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.929815 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.929942 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.931029 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:49.931033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:49.931036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:49.931048 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:49.931071 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:49.931084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:49.931155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:49.931320 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:00:49.931378 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:00:49.932583 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:49.924250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44245 (local address 127.0.98.2:39401)
0620 09:00:49.924939 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:49.924980 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:49.926263 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:49.926851 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:49.926858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:49.926939 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:49.927428 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.927455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.928323 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.928327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.929427 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:49.929437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:49.929989 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:49.929994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:49.930168 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:49.930776 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:49.930777 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:49.930816 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:49.930880 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:49.931168 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:49.931183 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:49.931189 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:00:49.931309 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:00:49.931597 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":160,"threads_started":1}
I20260620 09:00:49.933809 9987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44245
I20260620 09:00:49.934082 9987 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:49.934777 9987 heartbeater.cc:511] Master 127.0.98.62:44245 requested a full tablet report, sending...
I20260620 09:00:49.935932 9667 ts_manager.cc:194] Registered new tserver with Master: b6c45c1587f941ec94ad48ce76e6a88a (127.0.98.2:37363)
I20260620 09:00:49.936568 9667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39401
W20260620 09:00:50.045372 9990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:50.045658 9990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:50.045732 9990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:50.049839 9990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:50.049897 9990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:50.049993 9990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:50.054818 9990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:50.055917 9990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:50.057356 9990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:50.066884 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.066903 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.066884 9998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.067283 9990 server_base.cc:1034] running on GCE node
I20260620 09:00:50.067719 9990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:50.068359 9990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:50.069511 9990 hybrid_clock.cc:648] HybridClock initialized: now 1781946050069495 us; error 30 us; skew 500 ppm
I20260620 09:00:50.071242 9990 webserver.cc:460] Webserver started at http://127.0.98.3:33791/ using document root <none> and password file <none>
I20260620 09:00:50.071825 9990 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:50.071887 9990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:50.072124 9990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:50.074208 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "51a5d353718c4e67b1d0dca6e4f062a1"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "40f1ba8f658637e36e9a6a0bb01dcd46"
server_key_iv: "bd0f5ecfd9040e8b6bcf6aeacab7379f"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.074759 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51a5d353718c4e67b1d0dca6e4f062a1"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "40f1ba8f658637e36e9a6a0bb01dcd46"
server_key_iv: "bd0f5ecfd9040e8b6bcf6aeacab7379f"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.078581 9990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:50.081365 10004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.082597 9990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:50.082723 9990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "51a5d353718c4e67b1d0dca6e4f062a1"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "40f1ba8f658637e36e9a6a0bb01dcd46"
server_key_iv: "bd0f5ecfd9040e8b6bcf6aeacab7379f"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.082839 9990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:50.123487 9990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:50.124235 9990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:50.124482 9990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:50.125079 9990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:50.126034 9990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:50.126101 9990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.126168 9990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:50.126207 9990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.134533 9990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45061
I20260620 09:00:50.134595 10117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45061 every 8 connection(s)
I20260620 09:00:50.135483 9990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:50.137302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 9990
I20260620 09:00:50.137426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:50.137732 392 external_mini_cluster.cc:1428] Setting key 6adb90a54fac1dc944b040219a37e76c
I20260620 09:00:50.144966 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.137332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36375 (local address 127.0.98.62:44245)
0620 09:00:50.137484 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.137489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.138719 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 09:00:50.140576 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.140579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.140623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.140687 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.141561 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:50.142106 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:50.143214 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:50.143423 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:50.143498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:50.144541 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:50.144544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:50.144546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:50.144557 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:50.144576 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:50.144583 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:50.144641 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:50.144787 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:00:50.144838 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:00:50.145839 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.137803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44245 (local address 127.0.98.3:36375)
0620 09:00:50.138562 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:50.138614 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:50.140312 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:50.140829 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:50.140835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:50.140919 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:50.141435 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:50.141447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:50.142210 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:50.142213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:50.143106 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:50.143115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:50.143546 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:50.143548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:50.143689 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:50.144247 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:50.144254 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:50.144295 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:50.144360 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:50.144652 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:50.144659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:50.144663 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:50.144762 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 09:00:50.145001 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260620 09:00:50.147027 10118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44245
I20260620 09:00:50.147327 10118 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:50.147876 10118 heartbeater.cc:511] Master 127.0.98.62:44245 requested a full tablet report, sending...
I20260620 09:00:50.148928 9667 ts_manager.cc:194] Registered new tserver with Master: 51a5d353718c4e67b1d0dca6e4f062a1 (127.0.98.3:45061)
I20260620 09:00:50.149493 9667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36375
I20260620 09:00:50.151489 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:00:50.154235 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:50.153469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.0.98.62:44245)
0620 09:00:50.153636 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.153639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.153732 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:00:50.153813 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.153815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.153852 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.153915 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.154116 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48094: BlockingRecv error: recv got EOF from 127.0.0.1:48094 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:00:50.154399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9728
I20260620 09:00:50.160161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9859
I20260620 09:00:50.165325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9990
I20260620 09:00:50.170531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 9637
2026-06-20T09:00:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (917 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T09:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:50Z Disabled control of system clock
I20260620 09:00:50.199322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42991
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:00:50.309108 10129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:50.309378 10129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:50.309414 10129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:50.313372 10129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:50.313431 10129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:50.313446 10129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:50.313463 10129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:50.313478 10129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:50.318410 10129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:50.319520 10129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:50.320935 10129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:50.329303 10135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.329274 10137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.329311 10134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.329528 10129 server_base.cc:1034] running on GCE node
I20260620 09:00:50.330163 10129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:50.331319 10129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:50.332528 10129 hybrid_clock.cc:648] HybridClock initialized: now 1781946050332476 us; error 57 us; skew 500 ppm
I20260620 09:00:50.334347 10129 webserver.cc:460] Webserver started at http://127.0.98.62:41051/ using document root <none> and password file <none>
I20260620 09:00:50.334930 10129 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:50.334992 10129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:50.335240 10129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:50.337349 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "4b1138cdbf794637a3846f3c5812b6ff"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "413c8de753782dc80c200c55409d7297"
server_key_iv: "1e9df9c8aad1d29d515dc8001a48b715"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.337916 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b1138cdbf794637a3846f3c5812b6ff"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "413c8de753782dc80c200c55409d7297"
server_key_iv: "1e9df9c8aad1d29d515dc8001a48b715"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.341573 10129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 09:00:50.343950 10143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.345207 10129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:50.345342 10129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "4b1138cdbf794637a3846f3c5812b6ff"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "413c8de753782dc80c200c55409d7297"
server_key_iv: "1e9df9c8aad1d29d515dc8001a48b715"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.345455 10129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:50.350013 10129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:50.350610 10129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:50.350793 10129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:50.357575 10129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42991
I20260620 09:00:50.357596 10195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42991 every 8 connection(s)
I20260620 09:00:50.358489 10129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:50.361645 10196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:50.365118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10129
I20260620 09:00:50.365278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:50.365553 392 external_mini_cluster.cc:1428] Setting key 6b16a7cd795207e2260a267f6ab758bd
I20260620 09:00:50.367506 10196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff: Bootstrap starting.
I20260620 09:00:50.369882 10196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:50.370679 10196 log.cc:824] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:50.371205 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.366956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.0.98.62:42991)
0620 09:00:50.367614 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.367652 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.367699 (+ 47us) server_negotiation.cc:407] Connection header received
0620 09:00:50.368375 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.368390 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.368799 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.369126 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.369902 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:50.369906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:50.369917 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:50.369927 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:50.369998 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:50.370020 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:50.370120 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:50.370220 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 09:00:50.370452 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260620 09:00:50.372637 10196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff: No bootstrap required, opened a new log
I20260620 09:00:50.376299 10196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260620 09:00:50.376646 10196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:50.376715 10196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1138cdbf794637a3846f3c5812b6ff, State: Initialized, Role: FOLLOWER
I20260620 09:00:50.377172 10196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260620 09:00:50.377312 10196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:50.377374 10196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:50.377467 10196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:50.378387 10196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260620 09:00:50.378720 10196 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1138cdbf794637a3846f3c5812b6ff; no voters:
I20260620 09:00:50.379014 10196 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:50.379122 10201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:50.379392 10201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [term 1 LEADER]: Becoming Leader. State: Replica: 4b1138cdbf794637a3846f3c5812b6ff, State: Running, Role: LEADER
I20260620 09:00:50.379730 10201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260620 09:00:50.380288 10196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:50.381472 10203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1138cdbf794637a3846f3c5812b6ff. Latest consensus state: current_term: 1 leader_uuid: "4b1138cdbf794637a3846f3c5812b6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } } }
I20260620 09:00:50.381476 10202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1138cdbf794637a3846f3c5812b6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1138cdbf794637a3846f3c5812b6ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } } }
I20260620 09:00:50.381664 10202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:50.381980 10203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:50.382174 10210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:50.385339 10210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:50.390872 10210 catalog_manager.cc:1347] Generated new cluster ID: 54f186de3e88464eb77211147e4d0c14
I20260620 09:00:50.390964 10210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:50.416738 10210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:50.417120 10210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:50.425123 10210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1138cdbf794637a3846f3c5812b6ff: Generated new TSK 0
I20260620 09:00:50.425940 10210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:00:50.429083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:00:50.541944 10220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:50.542232 10220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:50.542307 10220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:50.546464 10220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:50.546528 10220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:50.546626 10220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:50.551667 10220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:50.552845 10220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:50.554288 10220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:50.563628 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.563721 10220 server_base.cc:1034] running on GCE node
W20260620 09:00:50.563628 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.563633 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.564335 10220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:50.565032 10220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:50.566252 10220 hybrid_clock.cc:648] HybridClock initialized: now 1781946050566235 us; error 37 us; skew 500 ppm
I20260620 09:00:50.568061 10220 webserver.cc:460] Webserver started at http://127.0.98.1:36955/ using document root <none> and password file <none>
I20260620 09:00:50.568720 10220 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:50.568813 10220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:50.569039 10220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:50.571105 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d7be037856a43879db330bc17114745"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "2050bbfc25ac74c06b389b288c84702f"
server_key_iv: "a29c7c04699705beca13eb5d775123bc"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.571679 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d7be037856a43879db330bc17114745"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "2050bbfc25ac74c06b389b288c84702f"
server_key_iv: "a29c7c04699705beca13eb5d775123bc"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.575338 10220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:00:50.577838 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.579047 10220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:50.579185 10220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "0d7be037856a43879db330bc17114745"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "2050bbfc25ac74c06b389b288c84702f"
server_key_iv: "a29c7c04699705beca13eb5d775123bc"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.579320 10220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:50.584255 10220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:50.584980 10220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:50.585165 10220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:50.585716 10220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:50.586637 10220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:50.586704 10220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.586763 10220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:50.586807 10220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.595332 10220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37967
I20260620 09:00:50.595353 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37967 every 8 connection(s)
I20260620 09:00:50.596262 10220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:50.602252 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.598141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56391 (local address 127.0.98.62:42991)
0620 09:00:50.598305 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.598309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.599379 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 09:00:50.600698 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.600701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.600740 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.600813 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.601813 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:50.601815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:50.601817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:50.601828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:50.601847 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:50.601854 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:50.601928 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:50.602099 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 09:00:50.602136 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:00:50.603188 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.598555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42991 (local address 127.0.98.1:56391)
0620 09:00:50.599204 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:50.599263 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:50.600480 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:50.600974 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:50.600980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:50.601066 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:50.601575 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:50.601576 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:50.601615 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:50.601681 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:50.601939 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:50.601946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:50.601951 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:00:50.602139 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 09:00:50.602399 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260620 09:00:50.604312 10348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42991
I20260620 09:00:50.604562 10348 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:50.604976 10348 heartbeater.cc:511] Master 127.0.98.62:42991 requested a full tablet report, sending...
I20260620 09:00:50.605129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10220
I20260620 09:00:50.605244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:00:50.605540 392 external_mini_cluster.cc:1428] Setting key 0a7a91d60f865eea4112b102a6ae5a05
I20260620 09:00:50.606621 10160 ts_manager.cc:194] Registered new tserver with Master: 0d7be037856a43879db330bc17114745 (127.0.98.1:37967)
I20260620 09:00:50.607510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:00:50.723169 10351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:50.723498 10351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:50.723634 10351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:50.732430 10351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:50.732539 10351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:50.732885 10351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:50.742654 10351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:50.744266 10351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:50.746356 10351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:50.757424 10357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.757552 10359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.757424 10356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.758414 10351 server_base.cc:1034] running on GCE node
I20260620 09:00:50.758968 10351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:50.759892 10351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:50.761091 10351 hybrid_clock.cc:648] HybridClock initialized: now 1781946050761068 us; error 39 us; skew 500 ppm
I20260620 09:00:50.763408 10351 webserver.cc:460] Webserver started at http://127.0.98.2:38225/ using document root <none> and password file <none>
I20260620 09:00:50.764148 10351 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:50.764230 10351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:50.764520 10351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:50.767553 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1ac930da4dc49af99e3fc6f298ac025"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "707590e26ec3de4a7691290548a2bc47"
server_key_iv: "c728758a1d6803b01d8844357719b503"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.768250 10351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1ac930da4dc49af99e3fc6f298ac025"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "707590e26ec3de4a7691290548a2bc47"
server_key_iv: "c728758a1d6803b01d8844357719b503"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.772689 10351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:00:50.775091 10365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.776286 10351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:00:50.776407 10351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "c1ac930da4dc49af99e3fc6f298ac025"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "707590e26ec3de4a7691290548a2bc47"
server_key_iv: "c728758a1d6803b01d8844357719b503"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.776558 10351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:50.781445 10351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:50.782037 10351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:50.782269 10351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:50.782824 10351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:50.783806 10351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:50.783854 10351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.783914 10351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:50.783948 10351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.792614 10351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37711
I20260620 09:00:50.792651 10478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37711 every 8 connection(s)
I20260620 09:00:50.793495 10351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:50.799247 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.795233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36799 (local address 127.0.98.62:42991)
0620 09:00:50.795373 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.795377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.796425 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 09:00:50.797753 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.797756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.797797 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.797870 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.798879 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:50.798882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:50.798884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:50.798894 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:50.798912 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:50.798919 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:50.798991 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:50.799096 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:00:50.799140 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:00:50.800094 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.795643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42991 (local address 127.0.98.2:36799)
0620 09:00:50.796286 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:50.796328 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:50.797566 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:50.798029 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:50.798036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:50.798123 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:50.798633 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:50.798635 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:50.798673 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:50.798739 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:50.798986 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:50.798991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:50.798995 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:00:50.799118 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 09:00:50.799310 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260620 09:00:50.801237 10479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42991
I20260620 09:00:50.801467 10479 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:50.801828 10479 heartbeater.cc:511] Master 127.0.98.62:42991 requested a full tablet report, sending...
I20260620 09:00:50.802754 10160 ts_manager.cc:194] Registered new tserver with Master: c1ac930da4dc49af99e3fc6f298ac025 (127.0.98.2:37711)
I20260620 09:00:50.803809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10351
I20260620 09:00:50.803926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:50.804239 392 external_mini_cluster.cc:1428] Setting key 5a5fbac844e9f4605cbb032f6288966d
I20260620 09:00:50.805970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:00:50.916163 10482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:50.916507 10482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:50.916601 10482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:50.920611 10482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:50.920724 10482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:50.920832 10482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:50.925760 10482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:50.926847 10482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:50.928179 10482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:50.937513 10490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.937506 10488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:50.937508 10487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:50.937682 10482 server_base.cc:1034] running on GCE node
I20260620 09:00:50.938364 10482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:50.938987 10482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:50.940140 10482 hybrid_clock.cc:648] HybridClock initialized: now 1781946050940139 us; error 45 us; skew 500 ppm
I20260620 09:00:50.941955 10482 webserver.cc:460] Webserver started at http://127.0.98.3:34875/ using document root <none> and password file <none>
I20260620 09:00:50.942504 10482 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:50.942561 10482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:50.942787 10482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:50.944893 10482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "87f681088bc247a89edef04828047695"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "f10c1d7e62b8da4e30db0161fbcda0a9"
server_key_iv: "217c8c94c4a4815023c4c0e5e9879aad"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.945441 10482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87f681088bc247a89edef04828047695"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "f10c1d7e62b8da4e30db0161fbcda0a9"
server_key_iv: "217c8c94c4a4815023c4c0e5e9879aad"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.949046 10482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:00:50.951450 10496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.952654 10482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:50.952776 10482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "87f681088bc247a89edef04828047695"
format_stamp: "Formatted at 2026-06-20 09:00:50 on dist-test-slave-scsf"
server_key: "f10c1d7e62b8da4e30db0161fbcda0a9"
server_key_iv: "217c8c94c4a4815023c4c0e5e9879aad"
server_key_version: "encryptionkey@0"
I20260620 09:00:50.952888 10482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:50.957988 10482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:50.958740 10482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:50.958930 10482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:50.959506 10482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:50.960395 10482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:50.960439 10482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.960544 10482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:50.960580 10482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:50.969009 10482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42201
I20260620 09:00:50.969069 10609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42201 every 8 connection(s)
I20260620 09:00:50.969941 10482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:50.971567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10482
I20260620 09:00:50.971679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:50.972004 392 external_mini_cluster.cc:1428] Setting key db2637544892f0641af12b4bd1e78a83
I20260620 09:00:50.976068 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.971725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46057 (local address 127.0.98.62:42991)
0620 09:00:50.971893 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.971899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.973191 (+ 1292us) server_negotiation.cc:407] Connection header received
0620 09:00:50.974680 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.974683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.974718 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.974784 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.975740 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:50.975743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:50.975745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:00:50.975755 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:50.975774 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:00:50.975781 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:50.975847 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:50.975937 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 09:00:50.975970 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:00:50.976984 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:50.972198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42991 (local address 127.0.98.3:46057)
0620 09:00:50.973049 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:50.973105 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:50.974489 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:50.974924 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:50.974930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:50.975016 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:50.975503 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:00:50.975505 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:50.975543 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:00:50.975607 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:50.975838 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:50.975842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:50.975845 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:00:50.975960 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:00:50.976140 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":174,"threads_started":1}
I20260620 09:00:50.978164 10610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42991
I20260620 09:00:50.978420 10610 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:50.978814 10610 heartbeater.cc:511] Master 127.0.98.62:42991 requested a full tablet report, sending...
I20260620 09:00:50.979825 10160 ts_manager.cc:194] Registered new tserver with Master: 87f681088bc247a89edef04828047695 (127.0.98.3:42201)
I20260620 09:00:50.985730 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:00:50.988628 10199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:00:50.987625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.0.98.62:42991)
0620 09:00:50.987788 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:50.987793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:50.987930 (+ 137us) server_negotiation.cc:407] Connection header received
0620 09:00:50.988058 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:50.988062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:50.988101 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:50.988183 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:50.988486 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51554: BlockingRecv error: recv got EOF from 127.0.0.1:51554 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 09:00:50.988819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10220
I20260620 09:00:50.994748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10351
I20260620 09:00:50.999883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10482
I20260620 09:00:51.005018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10129
2026-06-20T09:00:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (834 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:51 dist-test-slave-scsf krb5kdc[10620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:51 dist-test-slave-scsf krb5kdc[10620](info): set up 2 sockets
Jun 20 09:00:51 dist-test-slave-scsf krb5kdc[10620](info): commencing operation
krb5kdc: starting...
W20260620 09:00:52.033432 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:00:52.152990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:52.265631 10633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:52.265976 10633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:52.266052 10633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:52.270175 10633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:00:52.270292 10633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:52.270330 10633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:52.270364 10633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:00:52.270399 10633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:00:52.275868 10633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:52.277070 10633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:52.278463 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:52.287147 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.287127 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.287119 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:52.287413 10633 server_base.cc:1034] running on GCE node
I20260620 09:00:52.288053 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:52.289234 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:52.290431 10633 hybrid_clock.cc:648] HybridClock initialized: now 1781946052290418 us; error 26 us; skew 500 ppm
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:52.293232 10633 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:00:52.294199 10633 webserver.cc:460] Webserver started at http://127.0.98.62:38899/ using document root <none> and password file <none>
I20260620 09:00:52.294788 10633 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:52.294880 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:52.295115 10633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:52.297289 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "4e1dbf4b157c45e4bed078b045691260"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "a0e59a522fcccb0f881e544afa4b028e"
server_key_iv: "3355ced931d6cbe1536a229361a4a149"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.297874 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e1dbf4b157c45e4bed078b045691260"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "a0e59a522fcccb0f881e544afa4b028e"
server_key_iv: "3355ced931d6cbe1536a229361a4a149"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.301716 10633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:00:52.304172 10648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.305536 10633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:52.305718 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "4e1dbf4b157c45e4bed078b045691260"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "a0e59a522fcccb0f881e544afa4b028e"
server_key_iv: "3355ced931d6cbe1536a229361a4a149"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.305835 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:52.320698 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:52.323953 10633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:52.324119 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:52.331243 10633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36765
I20260620 09:00:52.331513 10700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36765 every 8 connection(s)
I20260620 09:00:52.332271 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:00:52.335129 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:52.339818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10633
I20260620 09:00:52.339965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:00:52.340238 392 external_mini_cluster.cc:1428] Setting key 8acfb07805e6e125a2347e60d06128a4
I20260620 09:00:52.340756 10701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: Bootstrap starting.
I20260620 09:00:52.343174 10701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:52.344108 10701 log.cc:824] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:52.346227 10701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: No bootstrap required, opened a new log
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:52.350670 10701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } }
I20260620 09:00:52.350934 10701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:52.350996 10701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1dbf4b157c45e4bed078b045691260, State: Initialized, Role: FOLLOWER
I20260620 09:00:52.351496 10701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } }
I20260620 09:00:52.351648 10701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:00:52.351727 10701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:00:52.351840 10701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:52.352825 10701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } }
I20260620 09:00:52.353169 10701 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1dbf4b157c45e4bed078b045691260; no voters:
I20260620 09:00:52.353525 10701 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:00:52.353600 10706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:52.353847 10706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1dbf4b157c45e4bed078b045691260, State: Running, Role: LEADER
I20260620 09:00:52.354213 10706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } }
I20260620 09:00:52.354924 10701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:00:52.355621 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:52.341614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.0.98.62:36765)
0620 09:00:52.342088 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:52.342105 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:52.342142 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:00:52.342780 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:52.342797 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:52.343143 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:52.343464 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:52.344391 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.345451 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.346581 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.347084 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.347212 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:52.349989 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:52.349995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:52.350011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:52.350037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:52.351935 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.352406 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.352412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.352418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.352587 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.352891 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.352896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.352898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.353380 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:52.353589 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:52.354355 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 09:00:52.354624 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":149,"threads_started":1}
I20260620 09:00:52.355854 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1dbf4b157c45e4bed078b045691260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } } }
I20260620 09:00:52.355996 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:00:52.356307 10707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1dbf4b157c45e4bed078b045691260. Latest consensus state: current_term: 1 leader_uuid: "4e1dbf4b157c45e4bed078b045691260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1dbf4b157c45e4bed078b045691260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36765 } } }
I20260620 09:00:52.356387 10707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260 [sys.catalog]: This master's current role is: LEADER
W20260620 09:00:52.359833 10722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:00:52.359931 10722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:00:52.360003 10715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:00:52.360802 10715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:00:52.366011 10715 catalog_manager.cc:1347] Generated new cluster ID: 38dea060f61a4bceb1b16e0c30589d79
I20260620 09:00:52.366098 10715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:00:52.383584 10715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:00:52.384658 10715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:00:52.392220 10715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: Generated new TSK 0
I20260620 09:00:52.393028 10715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:00:52.427749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:52.571410 10727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:52.571821 10727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:52.571895 10727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:52.576159 10727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:52.576246 10727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:52.576306 10727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:00:52.581712 10727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:52.582834 10727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:52.583859 10727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:52.593463 10732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.593463 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.593483 10735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:52.593708 10727 server_base.cc:1034] running on GCE node
I20260620 09:00:52.594291 10727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:52.594967 10727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:52.596139 10727 hybrid_clock.cc:648] HybridClock initialized: now 1781946052596118 us; error 32 us; skew 500 ppm
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:52.598941 10727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:00:52.599857 10727 webserver.cc:460] Webserver started at http://127.0.98.1:45055/ using document root <none> and password file <none>
I20260620 09:00:52.600437 10727 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:52.600580 10727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:52.600819 10727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:52.602898 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "fbb971b3fbbc4003a34c8cdddbf1629e"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "4c17d184066ca5722a1e1243e5721ac4"
server_key_iv: "3e2f33fc20b7161d054d76c3fea04b04"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.603507 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb971b3fbbc4003a34c8cdddbf1629e"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "4c17d184066ca5722a1e1243e5721ac4"
server_key_iv: "3e2f33fc20b7161d054d76c3fea04b04"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.607332 10727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 09:00:52.609803 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.611050 10727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:00:52.611189 10727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "fbb971b3fbbc4003a34c8cdddbf1629e"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "4c17d184066ca5722a1e1243e5721ac4"
server_key_iv: "3e2f33fc20b7161d054d76c3fea04b04"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.611297 10727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:52.621904 10727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:52.624814 10727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:52.625021 10727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:52.625599 10727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:52.626402 10727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:52.626467 10727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.626538 10727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:52.626585 10727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.635358 10727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39827
I20260620 09:00:52.635380 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39827 every 8 connection(s)
I20260620 09:00:52.636242 10727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:00:52.645218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10727
I20260620 09:00:52.645321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:52.645578 392 external_mini_cluster.cc:1428] Setting key 663dfbae2c468f5800343869cf5830ee
I20260620 09:00:52.650170 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:52.638013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46529 (local address 127.0.98.62:36765)
0620 09:00:52.638185 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:52.638190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:52.638997 (+ 807us) server_negotiation.cc:407] Connection header received
0620 09:00:52.640292 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:52.640295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:52.640342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:52.640420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:52.641622 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.642143 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.643212 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.643464 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.643541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:52.646174 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:52.646177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:52.646180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:52.646205 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:52.647813 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.648506 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.648509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.648511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.648595 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.648999 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.649002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.649004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.649235 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:52.649310 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:52.649943 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 09:00:52.650034 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 09:00:52.651621 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:52.638411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36765 (local address 127.0.98.1:46529)
0620 09:00:52.638851 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:52.638892 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:52.640080 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:52.640631 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:52.640639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:52.641003 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:52.641490 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:52.641504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.642243 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.642245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:52.643107 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:52.643115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.643588 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.643590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:52.643755 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:52.644382 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:52.644398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:52.645968 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:52.647941 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:52.647950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:52.647968 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:52.648359 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:52.648691 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:52.648696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:52.648699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:52.648900 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:52.649336 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:52.649346 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:52.649688 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 09:00:52.649914 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 09:00:52.650377 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":129,"threads_started":1}
I20260620 09:00:52.652966 10856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36765
I20260620 09:00:52.653223 10856 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:52.653708 10856 heartbeater.cc:511] Master 127.0.98.62:36765 requested a full tablet report, sending...
I20260620 09:00:52.655256 10665 ts_manager.cc:194] Registered new tserver with Master: fbb971b3fbbc4003a34c8cdddbf1629e (127.0.98.1:39827)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:00:52.656387 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:00:52.674823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:52.793865 10861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:52.794174 10861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:52.794261 10861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:52.798336 10861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:52.798424 10861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:52.798508 10861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:00:52.804591 10861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:52.805723 10861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:52.806756 10861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:52.816655 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.816682 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:52.816701 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:52.816877 10861 server_base.cc:1034] running on GCE node
I20260620 09:00:52.817330 10861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:52.818012 10861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:52.819201 10861 hybrid_clock.cc:648] HybridClock initialized: now 1781946052819168 us; error 48 us; skew 500 ppm
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:52.822021 10861 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:00:52.822947 10861 webserver.cc:460] Webserver started at http://127.0.98.2:45481/ using document root <none> and password file <none>
I20260620 09:00:52.823518 10861 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:52.823608 10861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:52.823833 10861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:52.825994 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "a84103a1dfcd4e77b48b664d5bd2f68c"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "e6abd86bcc14b876455a63de0e7e761d"
server_key_iv: "68f9f1f4ecdaaf15da1a04c79ba859b2"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.826622 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a84103a1dfcd4e77b48b664d5bd2f68c"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "e6abd86bcc14b876455a63de0e7e761d"
server_key_iv: "68f9f1f4ecdaaf15da1a04c79ba859b2"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.830411 10861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:00:52.832959 10876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.834316 10861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:52.834460 10861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "a84103a1dfcd4e77b48b664d5bd2f68c"
format_stamp: "Formatted at 2026-06-20 09:00:52 on dist-test-slave-scsf"
server_key: "e6abd86bcc14b876455a63de0e7e761d"
server_key_iv: "68f9f1f4ecdaaf15da1a04c79ba859b2"
server_key_version: "encryptionkey@0"
I20260620 09:00:52.834600 10861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:52.859831 10861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:52.862859 10861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:52.863073 10861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:52.863683 10861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:52.864583 10861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:52.864648 10861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.864720 10861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:52.864761 10861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:52.874059 10861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43979
I20260620 09:00:52.874078 10989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43979 every 8 connection(s)
I20260620 09:00:52.874989 10861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:00:52.881709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10861
I20260620 09:00:52.881807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:00:52.882249 392 external_mini_cluster.cc:1428] Setting key cc81f241e63e925c6f7049f424545c37
Jun 20 09:00:52 dist-test-slave-scsf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:52.889477 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:52.876818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47967 (local address 127.0.98.62:36765)
0620 09:00:52.876946 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:52.876950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:52.877738 (+ 788us) server_negotiation.cc:407] Connection header received
0620 09:00:52.879084 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:52.879087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:52.879142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:52.879229 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:52.880416 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.881007 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.882404 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.882686 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.882797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:52.885612 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:52.885615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:52.885617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:52.885641 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:52.887079 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.887673 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.887676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.887678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.887767 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:52.888213 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:52.888216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:52.888217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:52.888411 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:52.888571 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:52.889245 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 09:00:52.889353 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:00:52.890765 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:52.877167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36765 (local address 127.0.98.2:47967)
0620 09:00:52.877594 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:52.877637 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:52.878853 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:52.879416 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:52.879423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:52.879786 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:52.880268 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:52.880281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.881134 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.881140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:52.882277 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:52.882303 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:52.882857 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:52.882861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:52.883063 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:52.883745 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:52.883767 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:52.885428 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:52.887217 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:52.887221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:52.887235 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:52.887566 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:52.887860 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:52.887864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:52.887866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:52.888090 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:52.888563 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:52.888572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:52.888965 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 09:00:52.889215 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 09:00:52.889600 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:00:52.892045 10990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36765
I20260620 09:00:52.892297 10990 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:52.892839 10990 heartbeater.cc:511] Master 127.0.98.62:36765 requested a full tablet report, sending...
I20260620 09:00:52.893910 10665 ts_manager.cc:194] Registered new tserver with Master: a84103a1dfcd4e77b48b664d5bd2f68c (127.0.98.2:43979)
I20260620 09:00:52.894526 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:00:52.912073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:00:53.026904 10995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:00:53.027231 10995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:00:53.027309 10995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:00:53.031508 10995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:00:53.031623 10995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:00:53.031718 10995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:00:53.037761 10995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:00:53.039016 10995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:00:53.040127 10995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:00:53.050171 11000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:53.050117 11001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:00:53.050113 11003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:00:53.050195 10995 server_base.cc:1034] running on GCE node
I20260620 09:00:53.050889 10995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:00:53.051506 10995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:00:53.052671 10995 hybrid_clock.cc:648] HybridClock initialized: now 1781946053052635 us; error 48 us; skew 500 ppm
Jun 20 09:00:53 dist-test-slave-scsf krb5kdc[10620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:00:53.060434 10995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:00:53.061460 10995 webserver.cc:460] Webserver started at http://127.0.98.3:44675/ using document root <none> and password file <none>
I20260620 09:00:53.062067 10995 fs_manager.cc:359] Metadata directory not provided
I20260620 09:00:53.062165 10995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:00:53.062388 10995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:00:53.064527 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "940805901cd846148418be1f149d6546"
format_stamp: "Formatted at 2026-06-20 09:00:53 on dist-test-slave-scsf"
server_key: "d8d4b6fb90be8d4ba3158bf96e45dcda"
server_key_iv: "ef6a3701254b17029c26d69f04bc7687"
server_key_version: "encryptionkey@0"
I20260620 09:00:53.065119 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "940805901cd846148418be1f149d6546"
format_stamp: "Formatted at 2026-06-20 09:00:53 on dist-test-slave-scsf"
server_key: "d8d4b6fb90be8d4ba3158bf96e45dcda"
server_key_iv: "ef6a3701254b17029c26d69f04bc7687"
server_key_version: "encryptionkey@0"
I20260620 09:00:53.068981 10995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:00:53.071411 11010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:53.072757 10995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:00:53.072901 10995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "940805901cd846148418be1f149d6546"
format_stamp: "Formatted at 2026-06-20 09:00:53 on dist-test-slave-scsf"
server_key: "d8d4b6fb90be8d4ba3158bf96e45dcda"
server_key_iv: "ef6a3701254b17029c26d69f04bc7687"
server_key_version: "encryptionkey@0"
I20260620 09:00:53.073019 10995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:00:53.102043 10995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:00:53.105213 10995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:00:53.105436 10995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:00:53.106045 10995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:00:53.106894 10995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:00:53.106994 10995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:53.107061 10995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:00:53.107098 10995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:00:53.116683 10995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36405
I20260620 09:00:53.116701 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36405 every 8 connection(s)
I20260620 09:00:53.117633 10995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:00:53.118791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 10995
I20260620 09:00:53.118933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:00:53.119259 392 external_mini_cluster.cc:1428] Setting key f2fe9cd1ba94a761893fa1d3446ff6f0
Jun 20 09:00:53 dist-test-slave-scsf krb5kdc[10620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:00:53.131995 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.119682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32925 (local address 127.0.98.62:36765)
0620 09:00:53.119836 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.119840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.120972 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 09:00:53.122441 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.122443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.122490 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.122558 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:53.123733 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.124260 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.125517 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.125734 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.125812 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.128398 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:53.128402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:53.128404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:53.128432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:53.129803 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:53.130392 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:53.130397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:53.130399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:53.130489 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:53.130872 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:53.130877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:53.130879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:53.131071 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:53.131194 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.131703 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.131827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 09:00:53.133149 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.120232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36765 (local address 127.0.98.3:32925)
0620 09:00:53.120819 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.120880 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.122234 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.122739 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.122746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.123108 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:00:53.123594 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.123606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.124368 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.124370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.125397 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.125407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.125945 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.125969 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.126143 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.126703 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:00:53.126719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:00:53.128213 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:00:53.129936 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:53.129942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:53.129959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:53.130275 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:53.130620 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:00:53.130627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:00:53.130629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:00:53.130768 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:00:53.131192 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:00:53.131199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:00:53.131458 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.131687 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.132033 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":177,"threads_started":1}
I20260620 09:00:53.134230 11124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36765
I20260620 09:00:53.134485 11124 heartbeater.cc:464] Registering TS with master...
I20260620 09:00:53.135016 11124 heartbeater.cc:511] Master 127.0.98.62:36765 requested a full tablet report, sending...
I20260620 09:00:53.136055 10665 ts_manager.cc:194] Registered new tserver with Master: 940805901cd846148418be1f149d6546 (127.0.98.3:36405)
I20260620 09:00:53.136691 10665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32925
I20260620 09:00:53.145545 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:00:53.154948 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.147507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.98.62:36765)
0620 09:00:53.147665 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.147670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.147842 (+ 172us) server_negotiation.cc:407] Connection header received
0620 09:00:53.147947 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.147949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.147991 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.148070 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:00:53.148849 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.149377 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.150177 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.150379 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.150458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.151756 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:00:53.151759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:00:53.151761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:00:53.151800 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:00:53.153369 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:53.153746 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:53.153748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:53.153749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:53.153822 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:00:53.154057 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:00:53.154059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:00:53.154060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:00:53.154268 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:00:53.154369 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.154638 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.154730 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:00:53.158016 10665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:00:53.161165 10665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:00:53.175776 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.169717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39827 (local address 127.0.0.1:38608)
0620 09:00:53.170540 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.170609 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.170742 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.171496 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.171503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.171527 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:53.171758 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.171766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.172883 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.172889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.174174 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.174184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175187 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.175297 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.175332 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.175481 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.175623 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":68,"thread_start_us":83,"threads_started":1}
I20260620 09:00:53.175783 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.169825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.0.98.1:39827)
0620 09:00:53.170701 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.170708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.170734 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:00:53.170949 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.170957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.171186 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.171322 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:53.171894 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.172759 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.174335 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175046 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175189 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.175344 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.175526 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.175623 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":123,"threads_started":1}
I20260620 09:00:53.176271 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.170450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43979 (local address 127.0.0.1:42190)
0620 09:00:53.170925 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.170938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.171012 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.171743 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.171746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.171763 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:53.171900 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.171906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.173367 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.173370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.174512 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.174521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175816 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.175951 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.175985 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.176051 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.176115 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":85,"spinlock_wait_cycles":2432,"thread_start_us":86,"threads_started":1}
I20260620 09:00:53.176723 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.169790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36405 (local address 127.0.0.1:53000)
0620 09:00:53.170864 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.170877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.171049 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.171635 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.171638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.171652 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:53.171861 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.171866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.173342 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.173345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.174637 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.174646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175816 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.176306 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.176433 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.176532 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.176593 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":481,"thread_start_us":108,"threads_started":1}
I20260620 09:00:53.176798 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.170966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.98.2:43979)
0620 09:00:53.171284 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.171297 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.171319 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:00:53.171394 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.171399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.171591 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.171768 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:53.172034 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.173221 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.174643 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175504 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.175750 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.176487 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.176636 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":138,"threads_started":1}
I20260620 09:00:53.177470 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.169962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.98.3:36405)
0620 09:00:53.170380 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.170394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.170978 (+ 584us) server_negotiation.cc:407] Connection header received
0620 09:00:53.171178 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.171183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.171356 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.171488 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:53.171959 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.173187 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.174764 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.175504 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.175612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.175750 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.177231 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.177331 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":122,"threads_started":1}
I20260620 09:00:53.179123 10790 tablet_service.cc:1467] Processing CreateTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 (DEFAULT_TABLE table=test-table [id=b5b33cfaf7c949839857d75962562441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:53.180115 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613c9e02784e4aec927c9a6df48c55b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:53.179993 11058 tablet_service.cc:1467] Processing CreateTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 (DEFAULT_TABLE table=test-table [id=b5b33cfaf7c949839857d75962562441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:53.180997 11058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613c9e02784e4aec927c9a6df48c55b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:53.186914 11141 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Bootstrap starting.
I20260620 09:00:53.186832 10924 tablet_service.cc:1467] Processing CreateTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 (DEFAULT_TABLE table=test-table [id=b5b33cfaf7c949839857d75962562441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:00:53.187978 10924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613c9e02784e4aec927c9a6df48c55b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:00:53.188874 11142 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Bootstrap starting.
I20260620 09:00:53.189355 11141 tablet_bootstrap.cc:654] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:53.190297 11141 log.cc:824] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:53.191107 11142 tablet_bootstrap.cc:654] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:53.192469 11142 log.cc:824] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:53.192574 11141 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: No bootstrap required, opened a new log
I20260620 09:00:53.192778 11141 ts_tablet_manager.cc:1397] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 09:00:53.194356 11142 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: No bootstrap required, opened a new log
I20260620 09:00:53.194571 11142 ts_tablet_manager.cc:1397] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:00:53.195777 11145 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Bootstrap starting.
I20260620 09:00:53.197264 11141 raft_consensus.cc:348] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.197541 11141 raft_consensus.cc:374] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:53.197629 11141 raft_consensus.cc:729] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb971b3fbbc4003a34c8cdddbf1629e, State: Initialized, Role: FOLLOWER
I20260620 09:00:53.198125 11141 consensus_queue.cc:260] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.198434 11142 raft_consensus.cc:348] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.198832 11145 tablet_bootstrap.cc:654] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Neither blocks nor log segments found. Creating new log.
I20260620 09:00:53.198843 11142 raft_consensus.cc:374] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:53.198936 11142 raft_consensus.cc:729] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940805901cd846148418be1f149d6546, State: Initialized, Role: FOLLOWER
I20260620 09:00:53.199391 11141 ts_tablet_manager.cc:1428] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260620 09:00:53.199779 10856 heartbeater.cc:503] Master 127.0.98.62:36765 was elected leader, sending a full tablet report...
I20260620 09:00:53.199836 11145 log.cc:824] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:00:53.199574 11142 consensus_queue.cc:260] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.201521 11142 ts_tablet_manager.cc:1428] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:00:53.201921 11124 heartbeater.cc:503] Master 127.0.98.62:36765 was elected leader, sending a full tablet report...
I20260620 09:00:53.202080 11145 tablet_bootstrap.cc:492] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: No bootstrap required, opened a new log
I20260620 09:00:53.202275 11145 ts_tablet_manager.cc:1397] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:00:53.206007 11145 raft_consensus.cc:348] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.206254 11145 raft_consensus.cc:374] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:00:53.206288 11145 raft_consensus.cc:729] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84103a1dfcd4e77b48b664d5bd2f68c, State: Initialized, Role: FOLLOWER
I20260620 09:00:53.206636 11145 consensus_queue.cc:260] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.207223 10990 heartbeater.cc:503] Master 127.0.98.62:36765 was elected leader, sending a full tablet report...
I20260620 09:00:53.207576 11145 ts_tablet_manager.cc:1428] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:00:53.360661 11149 raft_consensus.cc:482] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:00:53.360931 11149 raft_consensus.cc:504] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.362205 11149 leader_election.cc:288] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb971b3fbbc4003a34c8cdddbf1629e (127.0.98.1:39827), 940805901cd846148418be1f149d6546 (127.0.98.3:36405)
I20260620 09:00:53.366279 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.362630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35625 (local address 127.0.98.1:39827)
0620 09:00:53.362757 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.362762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.362779 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:00:53.362839 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.362842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.362891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.362982 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:53.363370 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.363979 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365243 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.365679 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.365864 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.366075 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.366148 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:00:53.366291 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.362727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49397 (local address 127.0.98.3:36405)
0620 09:00:53.362875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.362879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.363104 (+ 225us) server_negotiation.cc:407] Connection header received
0620 09:00:53.363226 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.363229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.363287 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.363366 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:00:53.363650 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.364192 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365294 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.365702 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365831 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.365895 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.366118 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.366188 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:00:53.366282 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.362411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39827 (local address 127.0.98.2:35625)
0620 09:00:53.362567 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.362589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.362694 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.363003 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.363006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.363023 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:53.363212 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.363219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.364096 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.364099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.365080 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.365089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.365813 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.365926 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.365947 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.366014 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.366130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":94}
I20260620 09:00:53.366289 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.362588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36405 (local address 127.0.98.2:49397)
0620 09:00:53.362953 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.362969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:00:53.363110 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:00:53.363395 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:00:53.363398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:00:53.363415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:00:53.363525 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.363531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.364307 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.364310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.365136 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:00:53.365144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.365864 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.365868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:00:53.365964 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.365980 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:00:53.366084 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 09:00:53.366151 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
I20260620 09:00:53.367002 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613c9e02784e4aec927c9a6df48c55b0" candidate_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" is_pre_election: true
I20260620 09:00:53.367002 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613c9e02784e4aec927c9a6df48c55b0" candidate_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "940805901cd846148418be1f149d6546" is_pre_election: true
I20260620 09:00:53.367314 11078 raft_consensus.cc:2454] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84103a1dfcd4e77b48b664d5bd2f68c in term 0.
I20260620 09:00:53.367322 10810 raft_consensus.cc:2454] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84103a1dfcd4e77b48b664d5bd2f68c in term 0.
I20260620 09:00:53.367854 10877 leader_election.cc:302] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 940805901cd846148418be1f149d6546, a84103a1dfcd4e77b48b664d5bd2f68c, fbb971b3fbbc4003a34c8cdddbf1629e; no voters:
I20260620 09:00:53.368155 11149 raft_consensus.cc:2781] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:00:53.368240 11149 raft_consensus.cc:482] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:00:53.368283 11149 raft_consensus.cc:3037] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:00:53.368579 11125 tablet.cc:2369] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:53.369422 11149 raft_consensus.cc:504] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.369817 11149 leader_election.cc:288] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [CANDIDATE]: Term 1 election: Requested vote from peers fbb971b3fbbc4003a34c8cdddbf1629e (127.0.98.1:39827), 940805901cd846148418be1f149d6546 (127.0.98.3:36405)
I20260620 09:00:53.370190 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613c9e02784e4aec927c9a6df48c55b0" candidate_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "940805901cd846148418be1f149d6546"
I20260620 09:00:53.370223 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613c9e02784e4aec927c9a6df48c55b0" candidate_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e"
I20260620 09:00:53.370294 11078 raft_consensus.cc:3037] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:53.370378 10810 raft_consensus.cc:3037] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:00:53.371340 11078 raft_consensus.cc:2454] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84103a1dfcd4e77b48b664d5bd2f68c in term 1.
I20260620 09:00:53.371444 10810 raft_consensus.cc:2454] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84103a1dfcd4e77b48b664d5bd2f68c in term 1.
I20260620 09:00:53.371692 10877 leader_election.cc:302] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 940805901cd846148418be1f149d6546, a84103a1dfcd4e77b48b664d5bd2f68c; no voters:
I20260620 09:00:53.371891 11149 raft_consensus.cc:2781] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:00:53.372133 11149 raft_consensus.cc:686] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 1 LEADER]: Becoming Leader. State: Replica: a84103a1dfcd4e77b48b664d5bd2f68c, State: Running, Role: LEADER
I20260620 09:00:53.372509 11149 consensus_queue.cc:237] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } }
I20260620 09:00:53.375742 10664 catalog_manager.cc:5495] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c reported cstate change: term changed from 0 to 1, leader changed from <none> to a84103a1dfcd4e77b48b664d5bd2f68c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a84103a1dfcd4e77b48b664d5bd2f68c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43979 } health_report { overall_health: HEALTHY } } }
W20260620 09:00:53.376075 10991 tablet.cc:2369] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:00:53.387648 10857 tablet.cc:2369] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:00:53.412053 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.408617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.98.2:43979)
0620 09:00:53.408756 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:00:53.408760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:00:53.408773 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:00:53.408849 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:00:53.408852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:00:53.408907 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:00:53.409000 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:00:53.409237 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.409743 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.410625 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:00:53.410845 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:00:53.410923 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:00:53.411192 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:00:53.411641 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:00:53.411731 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:00:53.411790 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:00:53.411838 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:00:53.418262 11078 raft_consensus.cc:1261] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 1 FOLLOWER]: Refusing update from remote peer a84103a1dfcd4e77b48b664d5bd2f68c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:53.418262 10810 raft_consensus.cc:1261] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 1 FOLLOWER]: Refusing update from remote peer a84103a1dfcd4e77b48b664d5bd2f68c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:00:53.419080 11149 consensus_queue.cc:1035] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [LEADER]: Connected to new peer: Peer: permanent_uuid: "940805901cd846148418be1f149d6546" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:53.419299 11151 consensus_queue.cc:1035] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb971b3fbbc4003a34c8cdddbf1629e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:00:53.427776 11156 mvcc.cc:205] Tried to move back new op lower bound from 7298851034792968192 to 7298851034616676352. Current Snapshot: MvccSnapshot[applied={T|T < 7298851034792968192}]
I20260620 09:00:53.430823 11155 mvcc.cc:205] Tried to move back new op lower bound from 7298851034792968192 to 7298851034616676352. Current Snapshot: MvccSnapshot[applied={T|T < 7298851034792968192}]
I20260620 09:00:53.435415 10665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:53.435566 10665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:00:53.437934 10665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e1dbf4b157c45e4bed078b045691260: Sending DeleteTablet for 3 replicas of tablet 613c9e02784e4aec927c9a6df48c55b0
I20260620 09:00:53.438810 10790 tablet_service.cc:1514] Processing DeleteTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260620 09:00:53.438807 11058 tablet_service.cc:1514] Processing DeleteTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53000
I20260620 09:00:53.439157 10924 tablet_service.cc:1514] Processing DeleteTablet for tablet 613c9e02784e4aec927c9a6df48c55b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42190
I20260620 09:00:53.439595 11164 tablet_replica.cc:330] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e: stopping tablet replica
I20260620 09:00:53.439831 11165 tablet_replica.cc:330] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: stopping tablet replica
I20260620 09:00:53.439913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10727
I20260620 09:00:53.439971 11164 raft_consensus.cc:2229] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:53.440145 11165 raft_consensus.cc:2229] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:00:53.440291 11164 raft_consensus.cc:2258] T 613c9e02784e4aec927c9a6df48c55b0 P fbb971b3fbbc4003a34c8cdddbf1629e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:53.440577 11165 raft_consensus.cc:2258] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:53.440591 11163 tablet_replica.cc:330] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: stopping tablet replica
I20260620 09:00:53.441210 11163 raft_consensus.cc:2229] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:00:53.441509 11163 raft_consensus.cc:2258] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:00:53.442018 11165 ts_tablet_manager.cc:1905] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:53.442965 11163 ts_tablet_manager.cc:1905] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:00:53.445693 11163 ts_tablet_manager.cc:1918] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:53.445838 11163 log.cc:1196] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/613c9e02784e4aec927c9a6df48c55b0
I20260620 09:00:53.446235 11163 ts_tablet_manager.cc:1939] T 613c9e02784e4aec927c9a6df48c55b0 P 940805901cd846148418be1f149d6546: Deleting consensus metadata
I20260620 09:00:53.446260 11165 ts_tablet_manager.cc:1918] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:00:53.446501 11165 log.cc:1196] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/613c9e02784e4aec927c9a6df48c55b0
I20260620 09:00:53.446871 11165 ts_tablet_manager.cc:1939] T 613c9e02784e4aec927c9a6df48c55b0 P a84103a1dfcd4e77b48b664d5bd2f68c: Deleting consensus metadata
I20260620 09:00:53.447647 10649 catalog_manager.cc:4841] TS 940805901cd846148418be1f149d6546 (127.0.98.3:36405): tablet 613c9e02784e4aec927c9a6df48c55b0 (table test-table [id=b5b33cfaf7c949839857d75962562441]) successfully deleted
I20260620 09:00:53.448104 10652 catalog_manager.cc:4841] TS a84103a1dfcd4e77b48b664d5bd2f68c (127.0.98.2:43979): tablet 613c9e02784e4aec927c9a6df48c55b0 (table test-table [id=b5b33cfaf7c949839857d75962562441]) successfully deleted
W20260620 09:00:53.449210 10651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39827 (error 108)
I20260620 09:00:53.450060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10861
I20260620 09:00:53.450364 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:00:53.450039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39827 (local address 127.0.0.1:38616)
0620 09:00:53.450175 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:00:53.450247 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 09:00:53.451552 10651 catalog_manager.cc:4584] TS fbb971b3fbbc4003a34c8cdddbf1629e (127.0.98.1:39827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 613c9e02784e4aec927c9a6df48c55b0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39827: connect: Connection refused (error 111)
I20260620 09:00:53.456519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10995
I20260620 09:00:53.462076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 10633
2026-06-20T09:00:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2458 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:00:53 dist-test-slave-scsf krb5kdc[11169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:00:53 dist-test-slave-scsf krb5kdc[11169](info): set up 2 sockets
Jun 20 09:00:53 dist-test-slave-scsf krb5kdc[11169](info): commencing operation
krb5kdc: starting...
W20260620 09:00:54.517692 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:00:54 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:00:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:00:54.685488 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3Jagag/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:00:57.245 UTC [11199] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:00:57.245 UTC [11199] LOG: listening on IPv4 address "127.0.98.20", port 53467
2026-06-20 09:00:57.249 UTC [11199] LOG: listening on Unix socket "/tmp/.s.PGSQL.53467"
2026-06-20 09:00:57.273 UTC [11202] LOG: database system was shut down at 2026-06-20 09:00:55 UTC
2026-06-20 09:00:57.278 UTC [11199] LOG: database system is ready to accept connections
I20260620 09:00:58.225179 392 mini_postgres.cc:96] Postgres bound to 53467
2026-06-20 09:00:58.231 UTC [11210] FATAL: database "slave" does not exist
127.0.98.20:53467 - accepting connections
I20260620 09:00:58.232302 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:00:58.247146 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:00:58.721652 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:00:58.721745 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin
2026-06-20 09:00:58,987 [I] DB FLAVOR :POSTGRES
2026-06-20 09:00:58,988 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:00:58,988 [I] Checking connection..
2026-06-20 09:00:58,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:00:59,247 [I] Checking connection passed.
2026-06-20 09:00:59,248 [I] --------- Verifying version history table ---------
2026-06-20 09:00:59,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:00:59,502 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:00:59,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:00:59,756 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:00:59,757 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:00:59,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:01:00.014 UTC [11274] WARNING: there is no transaction in progress
2026-06-20 09:01:00,020 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:01:00,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:01:00,273 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:01:00,274 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:01:00,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:01:00,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:01:00,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-scsf.c.gcp-upstream.internal','N') ;"
2026-06-20 09:01:01,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:01:01,261 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:01:01,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:01:01,495 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:01:01,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:01:01,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:01:02,002 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:01:02,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:01:02.965 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:02.980 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:02.998 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.013 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.029 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.046 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.101 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.108 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.122 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.132 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.143 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.156 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.165 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.171 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.179 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03.186 UTC [11382] WARNING: there is no transaction in progress
2026-06-20 09:01:03,839 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:01:03,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:01:04,073 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:01:04,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:01:04,314 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:01:04,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:01:04,549 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:01:04,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:01:04,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:01:05,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-scsf.c.gcp-upstream.internal';"
2026-06-20 09:01:05,278 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:01:05,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:01:05,509 [I] DB_PATCHES have already been applied
I20260620 09:01:05.515247 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3Jagag/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:01:05.515318 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:01:05.518759 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:32951
Jun 20, 2026 9:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:01:06.652076 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.420s user 0.005s sys 0.005s
Jun 20, 2026 9:01:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:01:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 9:01:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:01:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:01:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:01:28.282675 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:01:28 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946088, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:01:28.377897 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:01:28.388386 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:01:29.154536 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:01:29.154618 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms
I20260620 09:01:29.154701 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53467
2026-06-20 09:01:29,348 [I] DB FLAVOR :POSTGRES
2026-06-20 09:01:29,348 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:01:29,348 [I] Checking connection
2026-06-20 09:01:29,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:01:29,581 [I] connection success
2026-06-20 09:01:29,582 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:01:29,582 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:01:29,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:01:29,844 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:01:29,844 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:01:29,844 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:01:29,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:01:30,128 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:01:30.584800 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:01:30.776208 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:01:30.819561 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:01:30 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946088, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:31.235461 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:01:31.235625 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:01:31.235652 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:01:31.239809 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11562
I20260620 09:01:31.239929 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:57295
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:01:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:01:33.502914 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.125s user 0.011s sys 0.166s
I20260620 09:01:33.503051 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:01:33.503153 392 mini_ranger_kms.cc:401] 127.0.98.20:57295/kms/v1/keys
Jun 20 09:01:33 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:01:34 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:01:34 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:01:36 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:01:36.795730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41019
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57295/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:01:36.909874 11613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:01:36.910218 11613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:01:36.910307 11613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:01:36.914407 11613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:01:36.914469 11613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:01:36.914489 11613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:01:36.914503 11613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:01:36.914518 11613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:01:36.920126 11613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:57295/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41019
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:01:36.921262 11613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:01:36.922287 11613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:01:36.931658 11621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:36.931591 11618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:36.931599 11613 server_base.cc:1034] running on GCE node
W20260620 09:01:36.931618 11619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:36.932714 11613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:01:36.933897 11613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:01:36.935092 11613 hybrid_clock.cc:648] HybridClock initialized: now 1781946096935079 us; error 37 us; skew 500 ppm
Jun 20 09:01:36 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:01:36.942339 11613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:01:36.943379 11613 webserver.cc:460] Webserver started at http://127.0.98.62:38601/ using document root <none> and password file <none>
I20260620 09:01:36.943934 11613 fs_manager.cc:359] Metadata directory not provided
I20260620 09:01:36.944015 11613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:01:36.944252 11613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:01:36 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:36.994293 11613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "bf18025442864d4c9cb7bc74eb1c1164"
format_stamp: "Formatted at 2026-06-20 09:01:36 on dist-test-slave-scsf"
server_key: "0e867ef9972d9e85ca7c0953bd5f72e3"
server_key_iv: "be27cfd89b6751b3aee31c62e0ef1936"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:36.994922 11613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf18025442864d4c9cb7bc74eb1c1164"
format_stamp: "Formatted at 2026-06-20 09:01:36 on dist-test-slave-scsf"
server_key: "0e867ef9972d9e85ca7c0953bd5f72e3"
server_key_iv: "be27cfd89b6751b3aee31c62e0ef1936"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:36.999423 11613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:01:37.013545 11628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:37.014972 11613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 09:01:37.015120 11613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "bf18025442864d4c9cb7bc74eb1c1164"
format_stamp: "Formatted at 2026-06-20 09:01:36 on dist-test-slave-scsf"
server_key: "0e867ef9972d9e85ca7c0953bd5f72e3"
server_key_iv: "be27cfd89b6751b3aee31c62e0ef1936"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:37.015261 11613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:01:37.038087 11613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:01:37.041319 11613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:01:37.041539 11613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:01:37.049551 11613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41019
I20260620 09:01:37.049578 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41019 every 8 connection(s)
I20260620 09:01:37.050650 11613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:01:37.053610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 11613
I20260620 09:01:37.053743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:01:37.054025 392 external_mini_cluster.cc:1428] Setting key 24ac54d3bd07b4afe0562379977558c9
I20260620 09:01:37.054103 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:01:37.061888 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: Bootstrap starting.
Jun 20 09:01:37 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:01:37.064582 11681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: Neither blocks nor log segments found. Creating new log.
I20260620 09:01:37.065769 11681 log.cc:824] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: Log is configured to *not* fsync() on all Append() calls
I20260620 09:01:37.068572 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:37.055386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.62:41019)
0620 09:01:37.055799 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:37.055821 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:37.056046 (+ 225us) server_negotiation.cc:407] Connection header received
0620 09:01:37.056987 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:37.057005 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:37.057358 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:37.057702 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:01:37.058500 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:37.059527 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:37.060406 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:37.060927 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:37.061111 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:37.063659 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:01:37.063664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:01:37.063677 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:01:37.063704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:01:37.065408 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:37.065856 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:37.065862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:37.065899 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:37.066028 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:37.066309 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:37.066313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:37.066314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:37.066710 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:01:37.066898 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:37.067238 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 09:01:37.067541 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260620 09:01:37.068744 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: No bootstrap required, opened a new log
I20260620 09:01:37.072106 11681 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260620 09:01:37.072345 11681 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:01:37.072434 11681 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf18025442864d4c9cb7bc74eb1c1164, State: Initialized, Role: FOLLOWER
I20260620 09:01:37.072940 11681 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260620 09:01:37.073087 11681 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:01:37.073154 11681 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:01:37.073248 11681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:01:37.074227 11681 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260620 09:01:37.074563 11681 leader_election.cc:302] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf18025442864d4c9cb7bc74eb1c1164; no voters:
I20260620 09:01:37.074911 11681 leader_election.cc:288] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:01:37.074994 11686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:01:37.075186 11686 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [term 1 LEADER]: Becoming Leader. State: Replica: bf18025442864d4c9cb7bc74eb1c1164, State: Running, Role: LEADER
I20260620 09:01:37.075853 11686 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } }
I20260620 09:01:37.076241 11681 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:01:37.077158 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf18025442864d4c9cb7bc74eb1c1164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } } }
I20260620 09:01:37.077286 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [sys.catalog]: This master's current role is: LEADER
I20260620 09:01:37.077585 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf18025442864d4c9cb7bc74eb1c1164. Latest consensus state: current_term: 1 leader_uuid: "bf18025442864d4c9cb7bc74eb1c1164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf18025442864d4c9cb7bc74eb1c1164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41019 } } }
I20260620 09:01:37.077690 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164 [sys.catalog]: This master's current role is: LEADER
I20260620 09:01:37.078624 11681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:01:37.079918 11695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:01:37.085822 11695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:01:37.093370 11695 catalog_manager.cc:1347] Generated new cluster ID: 6023ae17249f4505b863daaa93111d03
I20260620 09:01:37.093441 11695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:01:37.121820 11695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:01:37.123246 11695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:01:37.136716 11695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: Generated new TSK 0
I20260620 09:01:37.137763 11695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:01:37.843665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:01:38.008846 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:01:38.009186 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:01:38.009258 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:01:38.013561 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:01:38.013624 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:01:38.013685 11726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:01:38.019419 11726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:01:38.020612 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:01:38.021832 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:01:38.032003 11732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:38.032379 11731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:38.032519 11734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:38.033876 11726 server_base.cc:1034] running on GCE node
I20260620 09:01:38.034305 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:01:38.035141 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:01:38.036346 11726 hybrid_clock.cc:648] HybridClock initialized: now 1781946098036316 us; error 46 us; skew 500 ppm
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:01:38.039731 11726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:01:38.040913 11726 webserver.cc:460] Webserver started at http://127.0.98.1:37371/ using document root <none> and password file <none>
I20260620 09:01:38.041589 11726 fs_manager.cc:359] Metadata directory not provided
I20260620 09:01:38.041742 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:01:38.041929 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:38.067224 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "6460c9455f794cf9aa18f445d6a09b84"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "8290c349a5d03f44a2d5f8ae724aaeed"
server_key_iv: "5145fa9582a0e2dd74cd60381e67b50c"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.068010 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6460c9455f794cf9aa18f445d6a09b84"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "8290c349a5d03f44a2d5f8ae724aaeed"
server_key_iv: "5145fa9582a0e2dd74cd60381e67b50c"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.072726 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:01:38.095264 11743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.097038 11726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 09:01:38.097249 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "6460c9455f794cf9aa18f445d6a09b84"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "8290c349a5d03f44a2d5f8ae724aaeed"
server_key_iv: "5145fa9582a0e2dd74cd60381e67b50c"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.097397 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:01:38.123183 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:01:38.126910 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:01:38.127179 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:01:38.127962 11726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:01:38.129103 11726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:01:38.129210 11726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.129333 11726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:01:38.129410 11726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.140414 11726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43345
I20260620 09:01:38.140473 11857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43345 every 8 connection(s)
I20260620 09:01:38.141476 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:01:38.148160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 11726
I20260620 09:01:38.148298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:01:38.148758 392 external_mini_cluster.cc:1428] Setting key a8bae9638ffa156e88ffd284586084c7
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:01:38.161541 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.143886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53797 (local address 127.0.98.62:41019)
0620 09:01:38.144171 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.144176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.146118 (+ 1942us) server_negotiation.cc:407] Connection header received
0620 09:01:38.147321 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.147327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.147409 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.147528 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:01:38.149103 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.149937 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.153073 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.153447 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.153569 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.156784 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:01:38.156789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:01:38.156793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:01:38.156824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:01:38.158759 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.159476 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.159480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.159482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.159610 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.160107 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.160110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.160112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.160338 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:01:38.160423 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.161211 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.161350 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":69,"threads_started":1}
I20260620 09:01:38.163496 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.144268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41019 (local address 127.0.98.1:53797)
0620 09:01:38.145908 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.145977 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.147066 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.147781 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.147793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.148174 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:01:38.148909 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.148932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.151015 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.151019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.152925 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.152937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.153768 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.153772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.154002 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.154739 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:01:38.154763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:01:38.156568 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:01:38.158914 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.158921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.158941 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.159323 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.159711 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.159717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.159719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.159992 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.160528 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:01:38.160536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:01:38.160888 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.162008 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.162352 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":171,"threads_started":1}
I20260620 09:01:38.165292 11858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41019
I20260620 09:01:38.165539 11858 heartbeater.cc:464] Registering TS with master...
I20260620 09:01:38.166158 11858 heartbeater.cc:511] Master 127.0.98.62:41019 requested a full tablet report, sending...
I20260620 09:01:38.167950 11645 ts_manager.cc:194] Registered new tserver with Master: 6460c9455f794cf9aa18f445d6a09b84 (127.0.98.1:43345)
I20260620 09:01:38.169625 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53797
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:01:38.195209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:01:38.323741 11865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:01:38.324081 11865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:01:38.324193 11865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:01:38.328668 11865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:01:38.328755 11865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:01:38.328859 11865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:01:38.336221 11865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:01:38.337474 11865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:01:38.338665 11865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:01:38.349011 11872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:38.350694 11874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:38.350680 11865 server_base.cc:1034] running on GCE node
W20260620 09:01:38.350701 11871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:38.351295 11865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:01:38.352029 11865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:01:38.353220 11865 hybrid_clock.cc:648] HybridClock initialized: now 1781946098353162 us; error 74 us; skew 500 ppm
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:01:38.356137 11865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:01:38.357195 11865 webserver.cc:460] Webserver started at http://127.0.98.2:36217/ using document root <none> and password file <none>
I20260620 09:01:38.357834 11865 fs_manager.cc:359] Metadata directory not provided
I20260620 09:01:38.357925 11865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:01:38.358161 11865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:38.374540 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "1734ed5c64e744bdb676f77d5b1dc411"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "986443ac382f4230c0abcfc22a290a67"
server_key_iv: "22b57521f25c245539fade262f8b6cd7"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.375168 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1734ed5c64e744bdb676f77d5b1dc411"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "986443ac382f4230c0abcfc22a290a67"
server_key_iv: "22b57521f25c245539fade262f8b6cd7"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.380096 11865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 09:01:38.400051 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.401875 11865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:01:38.402027 11865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "1734ed5c64e744bdb676f77d5b1dc411"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "986443ac382f4230c0abcfc22a290a67"
server_key_iv: "22b57521f25c245539fade262f8b6cd7"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.402173 11865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:01:38.437858 11865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:01:38.441823 11865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:01:38.442126 11865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:01:38.442876 11865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:01:38.444053 11865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:01:38.444160 11865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.444264 11865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:01:38.444329 11865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.456966 11865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43395
I20260620 09:01:38.457170 11994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43395 every 8 connection(s)
I20260620 09:01:38.458323 11865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:01:38.465610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 11865
I20260620 09:01:38.465744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:01:38.466049 392 external_mini_cluster.cc:1428] Setting key b24e69861205681aea81e5e80003204d
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:01:38.476001 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.460593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59299 (local address 127.0.98.62:41019)
0620 09:01:38.460718 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.460721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.461875 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 09:01:38.463300 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.463304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.463366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.463476 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:01:38.465045 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.465887 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.467751 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.468129 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.468216 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.471502 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:01:38.471506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:01:38.471509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:01:38.471540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:01:38.473431 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.474128 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.474132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.474134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.474234 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.474654 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.474658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.474660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.474884 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:01:38.474976 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.475684 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.475818 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:01:38.478216 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.461130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41019 (local address 127.0.98.2:59299)
0620 09:01:38.461707 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.461763 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.463005 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.463708 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.463718 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.464123 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:01:38.464847 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.464869 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.466093 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.466097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.467598 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.467612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.468435 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.468440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.468702 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.469466 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:01:38.469489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:01:38.471287 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:01:38.473576 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.473582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.473601 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.473987 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.474341 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.474344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.474345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.474538 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.475042 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:01:38.475048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:01:38.475401 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.476710 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.477081 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":158,"threads_started":1}
I20260620 09:01:38.479842 11995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41019
I20260620 09:01:38.480098 11995 heartbeater.cc:464] Registering TS with master...
I20260620 09:01:38.480728 11995 heartbeater.cc:511] Master 127.0.98.62:41019 requested a full tablet report, sending...
I20260620 09:01:38.482164 11645 ts_manager.cc:194] Registered new tserver with Master: 1734ed5c64e744bdb676f77d5b1dc411 (127.0.98.2:43395)
I20260620 09:01:38.483114 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59299
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:01:38.523277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57295/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:01:38.697578 12000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:01:38.698038 12000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:01:38.698153 12000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:01:38.704183 12000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:01:38.704423 12000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:01:38.704597 12000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:01:38.713366 12000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:57295/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:01:38.715018 12000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:01:38.716414 12000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:01:38.728873 12006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:38.729449 12005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:01:38.729718 12008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:01:38.730808 12000 server_base.cc:1034] running on GCE node
I20260620 09:01:38.731408 12000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:01:38.732200 12000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:01:38.733429 12000 hybrid_clock.cc:648] HybridClock initialized: now 1781946098733329 us; error 93 us; skew 500 ppm
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:01:38.737224 12000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:01:38.738364 12000 webserver.cc:460] Webserver started at http://127.0.98.3:34795/ using document root <none> and password file <none>
I20260620 09:01:38.739064 12000 fs_manager.cc:359] Metadata directory not provided
I20260620 09:01:38.739144 12000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:01:38.739372 12000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:38.755817 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "ccf8b11a28ac4354b9244273e3898fb7"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "10d17d2f4c726bdd74e0f55b61436674"
server_key_iv: "781a1cb189325ed6c42a0b36616da398"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.756729 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ccf8b11a28ac4354b9244273e3898fb7"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "10d17d2f4c726bdd74e0f55b61436674"
server_key_iv: "781a1cb189325ed6c42a0b36616da398"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.773473 12000 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.002s sys 0.002s
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:01:38.794217 12015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.795889 12000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:01:38.796018 12000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "ccf8b11a28ac4354b9244273e3898fb7"
format_stamp: "Formatted at 2026-06-20 09:01:38 on dist-test-slave-scsf"
server_key: "10d17d2f4c726bdd74e0f55b61436674"
server_key_iv: "781a1cb189325ed6c42a0b36616da398"
server_key_version: "kuduclusterkey@0"
I20260620 09:01:38.796147 12000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:01:38.829859 12000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:01:38.833567 12000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:01:38.833801 12000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:01:38.834476 12000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:01:38.835501 12000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:01:38.835599 12000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.835675 12000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:01:38.835713 12000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:01:38.846235 12000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33661
I20260620 09:01:38.847370 12000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:01:38.853334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 12000
I20260620 09:01:38.853470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:01:38.853767 392 external_mini_cluster.cc:1428] Setting key 3afb5705665841f75ecadf714b694c5e
I20260620 09:01:38.855008 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33661 every 8 connection(s)
Jun 20 09:01:38 dist-test-slave-scsf krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:01:38.879976 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.857620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40529 (local address 127.0.98.62:41019)
0620 09:01:38.857746 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.857751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.860927 (+ 3176us) server_negotiation.cc:407] Connection header received
0620 09:01:38.862179 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.862183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.862240 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.862294 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:01:38.863992 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.864842 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.869989 (+ 5147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.870377 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.870464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.873941 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:01:38.873945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:01:38.873948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:01:38.873975 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:01:38.875644 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.876264 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.876268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.876269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.876351 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.876918 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.876922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.876924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.877173 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:01:38.877260 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.879684 (+ 2424us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.879813 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:01:38.881932 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.858180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41019 (local address 127.0.98.3:40529)
0620 09:01:38.860773 (+ 2593us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.860832 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.861922 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.862734 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.862744 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.863111 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:01:38.863817 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.863836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.865006 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.865009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.869821 (+ 4812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.869837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.870758 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.870763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.870980 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.871735 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:01:38.871757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:01:38.873722 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:01:38.875788 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.875794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.875813 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.876155 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.876490 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:01:38.876494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:01:38.876496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:01:38.876811 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:01:38.877326 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:01:38.877333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:01:38.879350 (+ 2017us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.880407 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.880827 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2360,"thread_start_us":217,"threads_started":1}
I20260620 09:01:38.883445 12129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41019
I20260620 09:01:38.883674 12129 heartbeater.cc:464] Registering TS with master...
I20260620 09:01:38.884203 12129 heartbeater.cc:511] Master 127.0.98.62:41019 requested a full tablet report, sending...
I20260620 09:01:38.885371 11645 ts_manager.cc:194] Registered new tserver with Master: ccf8b11a28ac4354b9244273e3898fb7 (127.0.98.3:33661)
I20260620 09:01:38.886071 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40529
I20260620 09:01:38.890893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:01:38.903311 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.893187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.0.98.62:41019)
0620 09:01:38.893296 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.893300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.893859 (+ 559us) server_negotiation.cc:407] Connection header received
0620 09:01:38.894036 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.894038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.894083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.894130 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:01:38.895095 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.895911 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.897219 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.897564 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.897632 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.899480 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:01:38.899484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:01:38.899486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:01:38.899515 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:01:38.901210 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.901821 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.901824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.901826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.901908 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:01:38.902183 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:01:38.902186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:01:38.902188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:01:38.902413 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:01:38.902524 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.903021 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.903140 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:01:38.907238 11645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:01:38.911649 11645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:01:38.929745 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.922153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33661 (local address 127.0.0.1:42786)
0620 09:01:38.924068 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.924081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.924169 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.925190 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.925196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.925218 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:01:38.925341 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.925347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.926695 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.926698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.927996 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.928007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.929167 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.929171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.929279 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.929326 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.929453 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.929582 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"thread_start_us":113,"threads_started":1}
I20260620 09:01:38.931173 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.922971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.3:33661)
0620 09:01:38.924474 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.924489 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.924515 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:01:38.924596 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.924601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.924831 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.924991 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:01:38.925488 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.926557 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.928176 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.929026 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.930152 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.930312 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.930368 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.931026 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":150,"threads_started":1}
I20260620 09:01:38.934760 12062 tablet_service.cc:1467] Processing CreateTablet for tablet 042b005f3b3049cb92bcad97f3032714 (DEFAULT_TABLE table=test-table [id=745db5738be240a4b2ad040306667728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:01:38.936039 12062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042b005f3b3049cb92bcad97f3032714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:01:38.940150 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.923005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43395 (local address 127.0.0.1:32936)
0620 09:01:38.927445 (+ 4440us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.927459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.927542 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.929291 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.929294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.929310 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:01:38.929396 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.929401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.937241 (+ 7840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.937245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.938453 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.938462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.939726 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.939730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.939819 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.939843 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.939909 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.939995 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4375,"thread_start_us":81,"threads_started":1}
I20260620 09:01:38.945070 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.922094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43345 (local address 127.0.0.1:40202)
0620 09:01:38.923645 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:38.923697 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:38.923823 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:38.934188 (+ 10365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:38.934191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:38.934209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:01:38.934311 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.934317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.940647 (+ 6330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.940650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.941808 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:38.941819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.944651 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.944655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:38.944759 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.944786 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:01:38.944835 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:01:38.944909 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"thread_start_us":121,"threads_started":1}
I20260620 09:01:38.945992 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.923613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.98.1:43345)
0620 09:01:38.932888 (+ 9275us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.932895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.932919 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:01:38.933206 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.933211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.933410 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.934475 (+ 1065us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:01:38.934608 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.936176 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.941977 (+ 5801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.942784 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.945538 (+ 2754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.945695 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.945746 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.945844 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9164,"thread_start_us":71,"threads_started":1}
I20260620 09:01:38.946044 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:38.923157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.0.98.2:43395)
0620 09:01:38.928671 (+ 5514us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:38.928686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:38.928710 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:01:38.928783 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:38.928788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:38.929007 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:38.929152 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:01:38.933688 (+ 4536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.937085 (+ 3397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.938612 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:38.939419 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:38.939526 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:38.939664 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:38.945701 (+ 6037us) server_negotiation.cc:299] Negotiation successful
0620 09:01:38.945860 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5386,"thread_start_us":85,"threads_started":1}
I20260620 09:01:38.949638 11925 tablet_service.cc:1467] Processing CreateTablet for tablet 042b005f3b3049cb92bcad97f3032714 (DEFAULT_TABLE table=test-table [id=745db5738be240a4b2ad040306667728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:01:38.949836 11791 tablet_service.cc:1467] Processing CreateTablet for tablet 042b005f3b3049cb92bcad97f3032714 (DEFAULT_TABLE table=test-table [id=745db5738be240a4b2ad040306667728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:01:38.950863 11791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042b005f3b3049cb92bcad97f3032714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:01:38.950863 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042b005f3b3049cb92bcad97f3032714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:01:38.956907 12147 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Bootstrap starting.
I20260620 09:01:38.960310 12149 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Bootstrap starting.
I20260620 09:01:38.960526 12147 tablet_bootstrap.cc:654] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Neither blocks nor log segments found. Creating new log.
I20260620 09:01:38.961501 12147 log.cc:824] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:01:38.963568 12149 tablet_bootstrap.cc:654] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Neither blocks nor log segments found. Creating new log.
I20260620 09:01:38.964895 12149 log.cc:824] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Log is configured to *not* fsync() on all Append() calls
I20260620 09:01:38.965387 12148 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: Bootstrap starting.
I20260620 09:01:38.967662 12148 tablet_bootstrap.cc:654] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: Neither blocks nor log segments found. Creating new log.
I20260620 09:01:38.968489 12148 log.cc:824] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: Log is configured to *not* fsync() on all Append() calls
I20260620 09:01:38.971170 12147 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: No bootstrap required, opened a new log
I20260620 09:01:38.972462 12147 ts_tablet_manager.cc:1397] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20260620 09:01:38.976999 12147 raft_consensus.cc:348] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:38.979472 12149 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: No bootstrap required, opened a new log
I20260620 09:01:38.979777 12149 ts_tablet_manager.cc:1397] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20260620 09:01:38.985317 12149 raft_consensus.cc:348] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:38.985729 12149 raft_consensus.cc:374] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:01:38.985987 12149 raft_consensus.cc:729] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1734ed5c64e744bdb676f77d5b1dc411, State: Initialized, Role: FOLLOWER
I20260620 09:01:38.986616 12149 consensus_queue.cc:260] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:38.988310 12149 ts_tablet_manager.cc:1428] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260620 09:01:38.989951 12148 tablet_bootstrap.cc:492] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: No bootstrap required, opened a new log
I20260620 09:01:38.990218 12148 ts_tablet_manager.cc:1397] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20260620 09:01:38.995937 11995 heartbeater.cc:503] Master 127.0.98.62:41019 was elected leader, sending a full tablet report...
I20260620 09:01:38.995222 12148 raft_consensus.cc:348] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:38.999351 12148 raft_consensus.cc:374] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:01:38.999441 12148 raft_consensus.cc:729] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6460c9455f794cf9aa18f445d6a09b84, State: Initialized, Role: FOLLOWER
I20260620 09:01:38.999783 12147 raft_consensus.cc:374] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:01:38.999951 12147 raft_consensus.cc:729] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf8b11a28ac4354b9244273e3898fb7, State: Initialized, Role: FOLLOWER
I20260620 09:01:39.000360 12148 consensus_queue.cc:260] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:39.001116 12147 consensus_queue.cc:260] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:39.003543 12148 ts_tablet_manager.cc:1428] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20260620 09:01:39.004004 12147 ts_tablet_manager.cc:1428] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260620 09:01:39.004243 11858 heartbeater.cc:503] Master 127.0.98.62:41019 was elected leader, sending a full tablet report...
I20260620 09:01:39.005925 12129 heartbeater.cc:503] Master 127.0.98.62:41019 was elected leader, sending a full tablet report...
I20260620 09:01:39.008929 12154 raft_consensus.cc:482] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:01:39.009141 12154 raft_consensus.cc:504] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:39.010416 12154 leader_election.cc:288] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccf8b11a28ac4354b9244273e3898fb7 (127.0.98.3:33661), 6460c9455f794cf9aa18f445d6a09b84 (127.0.98.1:43345)
I20260620 09:01:39.017211 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.011311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47773 (local address 127.0.98.3:33661)
0620 09:01:39.011434 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:39.011440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:39.012758 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 09:01:39.012952 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:39.012956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:39.013000 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:39.013046 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:01:39.013465 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.014210 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.015864 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.016455 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.016562 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:39.016639 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:39.017008 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 09:01:39.017081 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:01:39.017940 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.010579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33661 (local address 127.0.98.2:47773)
0620 09:01:39.012679 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:39.012693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:39.012824 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:39.013191 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:39.013193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:39.013209 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:01:39.013321 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:39.013327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.014377 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.014380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:39.015721 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:39.015732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.016752 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.016755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:39.016880 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:39.016902 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:01:39.017637 (+ 735us) client_negotiation.cc:240] Negotiation successful
0620 09:01:39.017728 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2019,"thread_start_us":112,"threads_started":1}
I20260620 09:01:39.018764 12079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042b005f3b3049cb92bcad97f3032714" candidate_uuid: "1734ed5c64e744bdb676f77d5b1dc411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8b11a28ac4354b9244273e3898fb7" is_pre_election: true
I20260620 09:01:39.019095 12079 raft_consensus.cc:2454] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1734ed5c64e744bdb676f77d5b1dc411 in term 0.
I20260620 09:01:39.019707 11883 leader_election.cc:302] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1734ed5c64e744bdb676f77d5b1dc411, ccf8b11a28ac4354b9244273e3898fb7; no voters:
I20260620 09:01:39.020068 12154 raft_consensus.cc:2781] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:01:39.020202 12154 raft_consensus.cc:482] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:01:39.020292 12154 raft_consensus.cc:3037] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:01:39.021767 12154 raft_consensus.cc:504] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:39.022547 12079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042b005f3b3049cb92bcad97f3032714" candidate_uuid: "1734ed5c64e744bdb676f77d5b1dc411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8b11a28ac4354b9244273e3898fb7"
I20260620 09:01:39.022673 12079 raft_consensus.cc:3037] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:01:39.024010 12079 raft_consensus.cc:2454] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1734ed5c64e744bdb676f77d5b1dc411 in term 1.
I20260620 09:01:39.024415 11883 leader_election.cc:302] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1734ed5c64e744bdb676f77d5b1dc411, ccf8b11a28ac4354b9244273e3898fb7; no voters:
I20260620 09:01:39.024688 12153 raft_consensus.cc:2781] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:01:39.024943 12153 raft_consensus.cc:686] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 1 LEADER]: Becoming Leader. State: Replica: 1734ed5c64e744bdb676f77d5b1dc411, State: Running, Role: LEADER
I20260620 09:01:39.025355 12153 consensus_queue.cc:237] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } }
I20260620 09:01:39.029186 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.011435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33259 (local address 127.0.98.1:43345)
0620 09:01:39.011540 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:39.011545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:39.018745 (+ 7200us) server_negotiation.cc:407] Connection header received
0620 09:01:39.018985 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:39.018989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:39.019042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:39.019090 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:01:39.020522 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.021303 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.027654 (+ 6351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.028369 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.028487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:39.028556 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:39.028961 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 09:01:39.029051 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:01:39.028944 11645 catalog_manager.cc:5495] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1734ed5c64e744bdb676f77d5b1dc411 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1734ed5c64e744bdb676f77d5b1dc411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1734ed5c64e744bdb676f77d5b1dc411" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43395 } health_report { overall_health: HEALTHY } } }
I20260620 09:01:39.029909 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.010966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43345 (local address 127.0.98.2:33259)
0620 09:01:39.018652 (+ 7686us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:39.018667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:39.018810 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:39.019223 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:01:39.019226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:01:39.019244 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:01:39.020355 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:39.020366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.021445 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.021448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:39.023161 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:01:39.023171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.028662 (+ 5491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.028666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:01:39.028782 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:39.028806 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:01:39.029654 (+ 848us) client_negotiation.cc:240] Negotiation successful
0620 09:01:39.029752 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7616,"thread_start_us":77,"threads_started":1}
I20260620 09:01:39.030850 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042b005f3b3049cb92bcad97f3032714" candidate_uuid: "1734ed5c64e744bdb676f77d5b1dc411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6460c9455f794cf9aa18f445d6a09b84" is_pre_election: true
I20260620 09:01:39.031175 11811 raft_consensus.cc:2454] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1734ed5c64e744bdb676f77d5b1dc411 in term 0.
I20260620 09:01:39.031929 12154 leader_election.cc:288] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [CANDIDATE]: Term 1 election: Requested vote from peers ccf8b11a28ac4354b9244273e3898fb7 (127.0.98.3:33661), 6460c9455f794cf9aa18f445d6a09b84 (127.0.98.1:43345)
I20260620 09:01:39.032313 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042b005f3b3049cb92bcad97f3032714" candidate_uuid: "1734ed5c64e744bdb676f77d5b1dc411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6460c9455f794cf9aa18f445d6a09b84"
I20260620 09:01:39.032428 11811 raft_consensus.cc:3037] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:01:39.033841 11811 raft_consensus.cc:2454] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1734ed5c64e744bdb676f77d5b1dc411 in term 1.
I20260620 09:01:39.054731 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.049696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.98.2:43395)
0620 09:01:39.049858 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:01:39.049863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:01:39.049880 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:01:39.049972 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:01:39.049975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:01:39.050044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:01:39.050172 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:01:39.050485 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.051302 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.052582 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:01:39.052915 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:01:39.053013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:01:39.053436 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:01:39.053997 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:01:39.054113 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:01:39.054534 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 09:01:39.054596 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:01:39.062037 11811 raft_consensus.cc:1261] T 042b005f3b3049cb92bcad97f3032714 P 6460c9455f794cf9aa18f445d6a09b84 [term 1 FOLLOWER]: Refusing update from remote peer 1734ed5c64e744bdb676f77d5b1dc411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:01:39.063019 12159 consensus_queue.cc:1035] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6460c9455f794cf9aa18f445d6a09b84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:01:39.067898 12079 raft_consensus.cc:1261] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 1 FOLLOWER]: Refusing update from remote peer 1734ed5c64e744bdb676f77d5b1dc411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:01:39.069336 12154 consensus_queue.cc:1035] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf8b11a28ac4354b9244273e3898fb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:01:39.071230 12162 mvcc.cc:205] Tried to move back new op lower bound from 7298851221748969472 to 7298851221611896832. Current Snapshot: MvccSnapshot[applied={T|T < 7298851221748969472}]
I20260620 09:01:39.081823 12163 mvcc.cc:205] Tried to move back new op lower bound from 7298851221748969472 to 7298851221611896832. Current Snapshot: MvccSnapshot[applied={T|T < 7298851221748969472}]
I20260620 09:01:39.082396 11645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:01:39.082552 11645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:01:39.085107 11645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf18025442864d4c9cb7bc74eb1c1164: Sending DeleteTablet for 3 replicas of tablet 042b005f3b3049cb92bcad97f3032714
I20260620 09:01:39.085856 12062 tablet_service.cc:1514] Processing DeleteTablet for tablet 042b005f3b3049cb92bcad97f3032714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:01:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42786
I20260620 09:01:39.087101 11925 tablet_service.cc:1514] Processing DeleteTablet for tablet 042b005f3b3049cb92bcad97f3032714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:01:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32936
I20260620 09:01:39.087555 11791 tablet_service.cc:1514] Processing DeleteTablet for tablet 042b005f3b3049cb92bcad97f3032714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:01:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40202
I20260620 09:01:39.087632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 11726
I20260620 09:01:39.100732 12171 tablet_replica.cc:330] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: stopping tablet replica
I20260620 09:01:39.101104 12171 raft_consensus.cc:2229] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:01:39.101475 12171 raft_consensus.cc:2258] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:01:39.102111 11631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43345 (error 108)
I20260620 09:01:39.103029 12170 tablet_replica.cc:330] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: stopping tablet replica
I20260620 09:01:39.103240 12171 ts_tablet_manager.cc:1905] T 042b005f3b3049cb92bcad97f3032714 P 1734ed5c64e744bdb676f77d5b1dc411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:01:39.103282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 11865
I20260620 09:01:39.105845 12170 raft_consensus.cc:2229] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:01:39.106280 12170 raft_consensus.cc:2258] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:01:39.107883 12170 ts_tablet_manager.cc:1905] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:01:39.110735 12170 ts_tablet_manager.cc:1918] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:01:39.111340 12170 log.cc:1196] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/042b005f3b3049cb92bcad97f3032714
I20260620 09:01:39.111774 12170 ts_tablet_manager.cc:1939] T 042b005f3b3049cb92bcad97f3032714 P ccf8b11a28ac4354b9244273e3898fb7: Deleting consensus metadata
I20260620 09:01:39.111892 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:01:39.111382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43345 (local address 127.0.0.1:40204)
0620 09:01:39.111737 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:39.111801 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
W20260620 09:01:39.112434 11631 catalog_manager.cc:4584] TS 6460c9455f794cf9aa18f445d6a09b84 (127.0.98.1:43345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 042b005f3b3049cb92bcad97f3032714: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43345: connect: Connection refused (error 111)
W20260620 09:01:39.112911 12142 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:01:39.111599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43395 (local address 127.0.0.1:32944)
0620 09:01:39.112040 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 09:01:39.112053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:01:39.112141 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:01:39.112833 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43395: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":391}
I20260620 09:01:39.113248 11630 catalog_manager.cc:4841] TS ccf8b11a28ac4354b9244273e3898fb7 (127.0.98.3:33661): tablet 042b005f3b3049cb92bcad97f3032714 (table test-table [id=745db5738be240a4b2ad040306667728]) successfully deleted
I20260620 09:01:39.113598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 12000
I20260620 09:01:39.125252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 11613
I20260620 09:01:39.135334 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:01:39.480742 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:01:39.480887 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:01:39.827135 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:01:39.827 UTC [11199] LOG: received smart shutdown request
2026-06-20 09:01:39.830 UTC [11199] LOG: background worker "logical replication launcher" (PID 11208) exited with exit code 1
2026-06-20 09:01:39.830 UTC [11203] LOG: shutting down
2026-06-20 09:01:39.983 UTC [11199] LOG: database system is shut down
2026-06-20T09:01:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46636 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:01:40 dist-test-slave-scsf krb5kdc[12182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:01:40 dist-test-slave-scsf krb5kdc[12182](info): set up 2 sockets
Jun 20 09:01:40 dist-test-slave-scsf krb5kdc[12182](info): commencing operation
krb5kdc: starting...
W20260620 09:01:41.141592 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:01:41 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:01:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:01:41.300853 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3Jagag/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:01:43.616 UTC [12212] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:01:43.616 UTC [12212] LOG: listening on IPv4 address "127.0.98.20", port 39747
2026-06-20 09:01:43.619 UTC [12212] LOG: listening on Unix socket "/tmp/.s.PGSQL.39747"
2026-06-20 09:01:43.640 UTC [12215] LOG: database system was shut down at 2026-06-20 09:01:42 UTC
2026-06-20 09:01:43.643 UTC [12212] LOG: database system is ready to accept connections
I20260620 09:01:44.592271 392 mini_postgres.cc:96] Postgres bound to 39747
2026-06-20 09:01:44.598 UTC [12223] FATAL: database "slave" does not exist
127.0.98.20:39747 - accepting connections
I20260620 09:01:44.599057 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:01:44.613719 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:01:45.046336 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:01:45.046435 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin
2026-06-20 09:01:45,319 [I] DB FLAVOR :POSTGRES
2026-06-20 09:01:45,320 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:01:45,320 [I] Checking connection..
2026-06-20 09:01:45,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:01:45,561 [I] Checking connection passed.
2026-06-20 09:01:45,561 [I] --------- Verifying version history table ---------
2026-06-20 09:01:45,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:01:45,800 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:01:45,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:01:46,061 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:01:46,061 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:01:46,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:01:46.307 UTC [12287] WARNING: there is no transaction in progress
2026-06-20 09:01:46,314 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:01:46,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:01:46,541 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:01:46,541 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:01:46,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:01:46,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:01:47,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-scsf.c.gcp-upstream.internal','N') ;"
2026-06-20 09:01:47,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:01:47,506 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:01:47,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:01:47,760 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:01:47,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:01:48,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:01:48,239 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:01:48,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:01:49.168 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.184 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.198 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.213 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.228 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.245 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.299 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.305 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.320 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.330 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.341 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.354 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.364 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.370 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.377 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49.383 UTC [12395] WARNING: there is no transaction in progress
2026-06-20 09:01:49,967 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:01:49,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:01:50,192 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:01:50,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:01:50,453 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:01:50,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:01:50,711 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:01:50,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:01:50,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:01:51,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-scsf.c.gcp-upstream.internal';"
2026-06-20 09:01:51,426 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:01:51,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:01:51,661 [I] DB_PATCHES have already been applied
I20260620 09:01:51.666827 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3Jagag/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:01:51.666908 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:01:51.670298 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:43955
Jun 20, 2026 9:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:01:52.770558 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.172s user 0.002s sys 0.007s
Jun 20, 2026 9:02:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:02:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:02:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:02:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:02:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:02:13.495997 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:02:13 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:02:13.591169 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:02:13.601073 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:02:14.402150 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:02:14.402241 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms
I20260620 09:02:14.402264 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39747
2026-06-20 09:02:14,572 [I] DB FLAVOR :POSTGRES
2026-06-20 09:02:14,572 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:02:14,572 [I] Checking connection
2026-06-20 09:02:14,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:02:14,824 [I] connection success
2026-06-20 09:02:14,825 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:02:14,825 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:02:14,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:02:15,067 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:02:15,067 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:02:15,067 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:02:15,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:02:15,313 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:02:15.770299 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:02:15.979368 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:02:16.024960 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:02:16 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:16.431165 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:02:16.431317 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task3Jagag/build/debug/bin/postgresql.jar:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3Jagag/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:02:16.431339 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:02:16.435518 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12575
I20260620 09:02:16.435604 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59649
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 09:02:18 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:18.638612 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.047s user 0.011s sys 0.142s
I20260620 09:02:18.638763 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:02:18.638841 392 mini_ranger_kms.cc:401] 127.0.98.20:59649/kms/v1/keys
Jun 20 09:02:18 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:02:19 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946138, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:02:21 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:02:21.645164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38071
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:21.764603 12626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:21.764871 12626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:21.764930 12626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:21.768916 12626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:21.768975 12626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:21.768993 12626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:21.769009 12626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:21.769055 12626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:21.774531 12626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59649/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38071
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:21.775529 12626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:21.776578 12626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:21.784736 12634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:21.784713 12632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:21.784792 12631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:21.785254 12626 server_base.cc:1034] running on GCE node
I20260620 09:02:21.785838 12626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:21.787088 12626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:21.788275 12626 hybrid_clock.cc:648] HybridClock initialized: now 1781946141788245 us; error 38 us; skew 500 ppm
Jun 20 09:02:21 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:21.791422 12626 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:21.792279 12626 webserver.cc:460] Webserver started at http://127.0.98.62:46445/ using document root <none> and password file <none>
I20260620 09:02:21.792862 12626 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:21.792938 12626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:21.793151 12626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:02:21 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:21.851382 12626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7"
format_stamp: "Formatted at 2026-06-20 09:02:21 on dist-test-slave-scsf"
server_key: "6e0b4fa75b530dc310fb74b49740a0f9"
server_key_iv: "19b24828f816846e335a5b019729549f"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:21.852051 12626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7"
format_stamp: "Formatted at 2026-06-20 09:02:21 on dist-test-slave-scsf"
server_key: "6e0b4fa75b530dc310fb74b49740a0f9"
server_key_iv: "19b24828f816846e335a5b019729549f"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:21.856156 12626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 09:02:21.873553 12641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:21.874970 12626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:02:21.875102 12626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7"
format_stamp: "Formatted at 2026-06-20 09:02:21 on dist-test-slave-scsf"
server_key: "6e0b4fa75b530dc310fb74b49740a0f9"
server_key_iv: "19b24828f816846e335a5b019729549f"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:21.875236 12626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:21.895992 12626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:21.899133 12626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:21.899353 12626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:21.907896 12626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38071
I20260620 09:02:21.907939 12693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38071 every 8 connection(s)
I20260620 09:02:21.908908 12626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:21.912288 12694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:21.912894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 12626
I20260620 09:02:21.913026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:02:21.913379 392 external_mini_cluster.cc:1428] Setting key 4421658d717927e93ad15e9ebd6a8ad3
I20260620 09:02:21.919900 12694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Bootstrap starting.
Jun 20 09:02:21 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:21.922843 12694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:21.924109 12694 log.cc:824] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:21.926399 12694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: No bootstrap required, opened a new log
I20260620 09:02:21.927205 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:21.914766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.0.98.62:38071)
0620 09:02:21.915171 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:21.915188 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:21.915241 (+ 53us) server_negotiation.cc:407] Connection header received
0620 09:02:21.915858 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:21.915874 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:21.916172 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:21.916515 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:21.917237 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:21.918242 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:21.919228 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:21.919798 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:21.919961 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:21.922474 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:21.922480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:21.922494 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:21.922521 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:21.924111 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:21.924552 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:21.924574 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:21.924581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:21.924721 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:21.925074 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:21.925079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:21.925080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:21.925518 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:21.925696 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:21.926044 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 09:02:21.926316 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":148,"threads_started":1}
I20260620 09:02:21.929821 12694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:21.930011 12694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:21.930059 12694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c3a51d38e64a4aa0f00c169e2e5fb7, State: Initialized, Role: FOLLOWER
I20260620 09:02:21.930495 12694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:21.930609 12694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:21.930692 12694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:21.930799 12694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:21.931684 12694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:21.932008 12694 leader_election.cc:302] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c3a51d38e64a4aa0f00c169e2e5fb7; no voters:
I20260620 09:02:21.932286 12694 leader_election.cc:288] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:02:21.932417 12699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:21.932674 12699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 29c3a51d38e64a4aa0f00c169e2e5fb7, State: Running, Role: LEADER
I20260620 09:02:21.933049 12699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:21.933466 12694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:21.934528 12699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c3a51d38e64a4aa0f00c169e2e5fb7. Latest consensus state: current_term: 1 leader_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } }
I20260620 09:02:21.935098 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } }
I20260620 09:02:21.935228 12699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:21.935307 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:21.935775 12694 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:02:21.936733 12713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:21.942902 12713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:21.950251 12713 catalog_manager.cc:1347] Generated new cluster ID: 58de34f7480e48db97f643f8c6c023bd
I20260620 09:02:21.950320 12713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:21.973585 12713 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:02:21.978274 12713 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:21.992290 12713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Generated new TSK 0
I20260620 09:02:21.993251 12713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:02:22.748106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:22.887573 12739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:22.887969 12739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:22.888070 12739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:22.893709 12739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:22.893850 12739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:22.893983 12739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:22.899896 12739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:22.900990 12739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:22.902014 12739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:22.915356 12745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:22.915834 12747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:22.916097 12744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:22.917408 12739 server_base.cc:1034] running on GCE node
I20260620 09:02:22.917857 12739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:22.918558 12739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:22.924295 12739 hybrid_clock.cc:648] HybridClock initialized: now 1781946142924225 us; error 78 us; skew 500 ppm
Jun 20 09:02:22 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:22.928210 12739 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:22.929329 12739 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260620 09:02:22.930123 12739 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:22.930260 12739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:22.930552 12739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:02:22 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:22.952973 12739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "349d47d17d314302ad30f0f7a5223e27"
format_stamp: "Formatted at 2026-06-20 09:02:22 on dist-test-slave-scsf"
server_key: "ccfd7c6fdd4cf6d15c9e02682f02fb7e"
server_key_iv: "31fe19f1264a565062cc50f2ddabb6d7"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:22.953744 12739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "349d47d17d314302ad30f0f7a5223e27"
format_stamp: "Formatted at 2026-06-20 09:02:22 on dist-test-slave-scsf"
server_key: "ccfd7c6fdd4cf6d15c9e02682f02fb7e"
server_key_iv: "31fe19f1264a565062cc50f2ddabb6d7"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:22.960036 12739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 09:02:22.980819 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:22.982714 12739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:02:22.982859 12739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "349d47d17d314302ad30f0f7a5223e27"
format_stamp: "Formatted at 2026-06-20 09:02:22 on dist-test-slave-scsf"
server_key: "ccfd7c6fdd4cf6d15c9e02682f02fb7e"
server_key_iv: "31fe19f1264a565062cc50f2ddabb6d7"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:22.983011 12739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:23.016178 12739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:23.019804 12739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:23.020099 12739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:23.020900 12739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:23.021998 12739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:23.022110 12739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.022233 12739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:23.022315 12739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.035578 12739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35895
I20260620 09:02:23.036842 12739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:23.037447 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35895 every 8 connection(s)
I20260620 09:02:23.044277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 12739
I20260620 09:02:23.044400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:02:23.044831 392 external_mini_cluster.cc:1428] Setting key e6d75645f766dcfb76b428420528d154
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:02:23.059288 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.040252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.1:54267)
0620 09:02:23.044439 (+ 4187us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:23.044518 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:23.045544 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:23.046658 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:23.046667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:23.047010 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:23.047643 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.047679 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.048789 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.048793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.050193 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.050205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.050870 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.050874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.051064 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.051749 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:23.051769 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:23.053716 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:23.056017 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.056022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.056040 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.056392 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.056748 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.056752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.056754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.056945 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.057393 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:23.057399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:23.057709 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 09:02:23.057883 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 09:02:23.058266 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3927,"thread_start_us":474,"threads_started":1}
I20260620 09:02:23.060299 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.040683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54267 (local address 127.0.98.62:38071)
0620 09:02:23.045982 (+ 5299us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:23.045986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:23.045999 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:02:23.046035 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:23.046038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:23.046078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:23.046161 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:23.047825 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.048641 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.050342 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.050699 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.051384 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.053906 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:23.053910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:23.053913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:23.053950 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:23.055877 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.056543 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.056546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.056548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.056640 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.057051 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.057054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.057055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.057280 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:23.059924 (+ 2644us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:23.060000 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:02:23.060129 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5216,"thread_start_us":63,"threads_started":1}
I20260620 09:02:23.061539 12871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:23.061787 12871 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:23.062312 12871 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:23.063968 12658 ts_manager.cc:194] Registered new tserver with Master: 349d47d17d314302ad30f0f7a5223e27 (127.0.98.1:35895)
I20260620 09:02:23.065490 12658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:02:23.085229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:02:23.235740 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:23.236109 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:23.236188 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:23.242245 12878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:23.242323 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:23.242416 12878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:23.250234 12878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:23.251696 12878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:23.253139 12878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:23.265513 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.265513 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.266041 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:23.266487 12878 server_base.cc:1034] running on GCE node
I20260620 09:02:23.266981 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:23.267765 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:23.268961 12878 hybrid_clock.cc:648] HybridClock initialized: now 1781946143268937 us; error 36 us; skew 500 ppm
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:23.272516 12878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:23.273701 12878 webserver.cc:460] Webserver started at http://127.0.98.2:41809/ using document root <none> and password file <none>
I20260620 09:02:23.274256 12878 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:23.274315 12878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:23.274530 12878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:23.293072 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ce0de8e50bc4c5396c6fcd60e829371"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "be599cd542da7d49315dd59412230ef0"
server_key_iv: "338425aa891ead2e46c78d51e1a7b45e"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.293906 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ce0de8e50bc4c5396c6fcd60e829371"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "be599cd542da7d49315dd59412230ef0"
server_key_iv: "338425aa891ead2e46c78d51e1a7b45e"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.299073 12878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 09:02:23.324541 12894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.326205 12878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:02:23.326360 12878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "0ce0de8e50bc4c5396c6fcd60e829371"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "be599cd542da7d49315dd59412230ef0"
server_key_iv: "338425aa891ead2e46c78d51e1a7b45e"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.326514 12878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:23.349419 12878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:23.353230 12878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:23.353438 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:23.354187 12878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:23.355316 12878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:23.355386 12878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.355463 12878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:23.355497 12878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.366501 12878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33899
I20260620 09:02:23.367744 12878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:02:23.372587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 12878
I20260620 09:02:23.372743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:02:23.373062 392 external_mini_cluster.cc:1428] Setting key 9473b6ff68f057631b77ffbe380924da
I20260620 09:02:23.378903 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33899 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:23.410293 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.382133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57493 (local address 127.0.98.62:38071)
0620 09:02:23.382271 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:23.382276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:23.387860 (+ 5584us) server_negotiation.cc:407] Connection header received
0620 09:02:23.391450 (+ 3590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:23.391454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:23.391520 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:23.391604 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:23.393429 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.394263 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.396003 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.396358 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.396436 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.404152 (+ 7716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:23.404158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:23.404161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:23.404195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:23.406020 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.406699 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.406703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.406705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.406794 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.408722 (+ 1928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.408726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.408728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.408956 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:23.409042 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:23.409883 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 09:02:23.410024 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:02:23.412492 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.380925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.2:57493)
0620 09:02:23.387607 (+ 6682us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:23.387721 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:23.391172 (+ 3451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:23.392117 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:23.392128 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:23.392534 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:23.393256 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.393277 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.394432 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.394435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.395848 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.395858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.396663 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.396667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.396880 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.400653 (+ 3773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:23.400679 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:23.403885 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:23.406176 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.406182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.406200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.406571 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.406913 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.406916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.406918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.407105 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.409110 (+ 2005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:23.409116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:23.409502 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 09:02:23.410505 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0620 09:02:23.411287 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6400,"thread_start_us":189,"threads_started":1}
I20260620 09:02:23.414024 13008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:23.414331 13008 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:23.414906 13008 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:23.415990 12658 ts_manager.cc:194] Registered new tserver with Master: 0ce0de8e50bc4c5396c6fcd60e829371 (127.0.98.2:33899)
I20260620 09:02:23.416806 12658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:02:23.423939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:23.589386 13013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:23.589751 13013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:23.589839 13013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:23.595942 13013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:23.596025 13013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:23.596114 13013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:23.603974 13013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:23.605410 13013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:23.606729 13013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:23.619942 13018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.619963 13019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.620412 13021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:23.621354 13013 server_base.cc:1034] running on GCE node
I20260620 09:02:23.621862 13013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:23.622661 13013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:23.623847 13013 hybrid_clock.cc:648] HybridClock initialized: now 1781946143623770 us; error 91 us; skew 500 ppm
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:23.627351 13013 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:23.628569 13013 webserver.cc:460] Webserver started at http://127.0.98.3:41877/ using document root <none> and password file <none>
I20260620 09:02:23.629284 13013 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:23.629375 13013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:23.629609 13013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:23.645238 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb64cf5e38d74c1884dede20f0633854"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "2ad706b7e447e02c6b436a8fa100afcb"
server_key_iv: "304eaa0bec881c7c0136c921d7d6c156"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.652525 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb64cf5e38d74c1884dede20f0633854"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "2ad706b7e447e02c6b436a8fa100afcb"
server_key_iv: "304eaa0bec881c7c0136c921d7d6c156"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.664472 13013 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260620 09:02:23.686735 13028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.690166 13013 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260620 09:02:23.690395 13013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "bb64cf5e38d74c1884dede20f0633854"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "2ad706b7e447e02c6b436a8fa100afcb"
server_key_iv: "304eaa0bec881c7c0136c921d7d6c156"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.690701 13013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:23.712107 13013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:23.715546 13013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:23.715718 13013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:23.716358 13013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:23.717309 13013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:23.717377 13013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.717446 13013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:23.717474 13013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.727121 13013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39569
I20260620 09:02:23.728204 13013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:23.729557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13013
I20260620 09:02:23.729669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:02:23.729969 392 external_mini_cluster.cc:1428] Setting key 00fd2c9dce6dca06416940a58b2a85e1
I20260620 09:02:23.742095 13141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39569 every 8 connection(s)
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:23.758155 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.745651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59477 (local address 127.0.98.62:38071)
0620 09:02:23.745779 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:23.745783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:23.746025 (+ 242us) server_negotiation.cc:407] Connection header received
0620 09:02:23.747195 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:23.747198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:23.747249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:23.747295 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:23.748876 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.749673 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.751254 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.751546 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.751632 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.754460 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:23.754464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:23.754467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:23.754490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:23.755967 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.756524 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.756528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.756530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.756603 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:23.756953 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:23.756956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:23.756958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:23.757159 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:23.757231 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:23.757829 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 09:02:23.757958 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:02:23.759853 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:23.744569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.3:59477)
0620 09:02:23.745876 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:23.745935 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:23.746954 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:23.747708 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:23.747717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:23.748065 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:23.748728 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.748745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.749820 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.749823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.751120 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:23.751138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:23.751820 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:23.751824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:23.752002 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:23.752696 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:23.752716 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:23.754292 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:23.756090 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.756095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.756110 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.756404 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.756698 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:23.756700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:23.756702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:23.756853 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:23.757291 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:23.757296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:23.757581 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 09:02:23.758565 (+ 984us) client_negotiation.cc:240] Negotiation successful
0620 09:02:23.758877 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":161,"threads_started":1}
I20260620 09:02:23.761236 13142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:23.761454 13142 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:23.762023 13142 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:23.763110 12658 ts_manager.cc:194] Registered new tserver with Master: bb64cf5e38d74c1884dede20f0633854 (127.0.98.3:39569)
I20260620 09:02:23.763818 12658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59477
I20260620 09:02:23.767184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:02:23.768010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 12739
I20260620 09:02:23.776396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 12878
I20260620 09:02:23.784540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13013
I20260620 09:02:23.792855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 12626
I20260620 09:02:23.800894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--webserver_interface=127.0.98.62
--webserver_port=46445
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38071
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:23.907826 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:23.908067 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:23.908164 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:23.912070 13145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:23.912123 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:23.912142 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:23.912189 13145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:23.912225 13145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:23.917279 13145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38071
--ranger_config_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:23.918244 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:23.919195 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:23.926889 13151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.926954 13150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:23.926954 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:23.927075 13145 server_base.cc:1034] running on GCE node
I20260620 09:02:23.927655 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:23.928692 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:23.929847 13145 hybrid_clock.cc:648] HybridClock initialized: now 1781946143929833 us; error 26 us; skew 500 ppm
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:23.932368 13145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:23.933204 13145 webserver.cc:460] Webserver started at http://127.0.98.62:46445/ using document root <none> and password file <none>
I20260620 09:02:23.933674 13145 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:23.933730 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:02:23 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:23.974721 13145 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 09:02:23.977063 13161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:23.978396 13145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:02:23.978533 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7"
format_stamp: "Formatted at 2026-06-20 09:02:21 on dist-test-slave-scsf"
server_key: "6e0b4fa75b530dc310fb74b49740a0f9"
server_key_iv: "19b24828f816846e335a5b019729549f"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:23.979106 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:24.002007 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:24.005065 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:24.005273 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:24.012256 13145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38071
I20260620 09:02:24.012271 13213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38071 every 8 connection(s)
I20260620 09:02:24.013302 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:24.017262 13214 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:02:24.018456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13145
I20260620 09:02:24.018889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35895
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38071
--webserver_port=37593
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:24.018891 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Bootstrap starting.
I20260620 09:02:24.034642 13214 log.cc:824] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:24.041113 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:24.041494 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Bootstrap complete.
I20260620 09:02:24.045542 13214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:24.045950 13214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c3a51d38e64a4aa0f00c169e2e5fb7, State: Initialized, Role: FOLLOWER
I20260620 09:02:24.046411 13214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:24.046550 13214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:24.046623 13214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:24.046733 13214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:02:24.048496 13214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:24.048815 13214 leader_election.cc:302] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c3a51d38e64a4aa0f00c169e2e5fb7; no voters:
I20260620 09:02:24.049110 13214 leader_election.cc:288] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:02:24.049213 13218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:02:24.049423 13218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [term 2 LEADER]: Becoming Leader. State: Replica: 29c3a51d38e64a4aa0f00c169e2e5fb7, State: Running, Role: LEADER
I20260620 09:02:24.049691 13218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } }
I20260620 09:02:24.050253 13214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:24.050591 13220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c3a51d38e64a4aa0f00c169e2e5fb7. Latest consensus state: current_term: 2 leader_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } }
I20260620 09:02:24.050747 13220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:24.050943 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c3a51d38e64a4aa0f00c169e2e5fb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } }
I20260620 09:02:24.051051 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:24.051397 13222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:24.054479 13222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:24.055013 13214 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:02:24.055061 13222 catalog_manager.cc:1259] Loaded cluster ID: 58de34f7480e48db97f643f8c6c023bd
I20260620 09:02:24.055189 13222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:24.059829 13222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:24.061239 13222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Loaded TSK: 0
I20260620 09:02:24.062772 13222 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:02:24.160651 13216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:24.160985 13216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:24.161060 13216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:24.167224 13216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:24.167304 13216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:24.167395 13216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:24.175416 13216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37593
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:24.176779 13216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:24.178092 13216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:24.190301 13254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:24.190330 13252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:24.192726 13251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:24.193073 13216 server_base.cc:1034] running on GCE node
I20260620 09:02:24.193544 13216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:24.194334 13216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:24.195526 13216 hybrid_clock.cc:648] HybridClock initialized: now 1781946144195496 us; error 42 us; skew 500 ppm
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:24.199229 13216 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:24.200405 13216 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260620 09:02:24.201145 13216 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:24.201285 13216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:24.225885 13216 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 09:02:24.228912 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.230567 13216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:02:24.230700 13216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "349d47d17d314302ad30f0f7a5223e27"
format_stamp: "Formatted at 2026-06-20 09:02:22 on dist-test-slave-scsf"
server_key: "ccfd7c6fdd4cf6d15c9e02682f02fb7e"
server_key_iv: "31fe19f1264a565062cc50f2ddabb6d7"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:24.231389 13216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:24.249400 13216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:24.252996 13216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:24.253247 13216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:24.253998 13216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:24.255044 13216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:24.255168 13216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.255303 13216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:24.255391 13216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.266124 13216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35895
I20260620 09:02:24.267405 13216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:24.268573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13216
I20260620 09:02:24.269032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38071
--webserver_port=41809
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:24.282079 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35895 every 8 connection(s)
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:24.322381 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.284827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50067 (local address 127.0.98.62:38071)
0620 09:02:24.292205 (+ 7378us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:24.292230 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:24.292285 (+ 55us) server_negotiation.cc:407] Connection header received
0620 09:02:24.292617 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:24.292644 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:24.293084 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:24.293432 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:24.294732 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.296282 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.301796 (+ 5514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.302529 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.302710 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.305691 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:24.305698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:24.305713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:24.305745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:24.316209 (+ 10464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.318464 (+ 2255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.318470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.318476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.318635 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.319088 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.319091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.319093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.319575 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:24.319711 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:24.321085 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0620 09:02:24.321399 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7202,"thread_start_us":114,"threads_started":1}
I20260620 09:02:24.324112 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.283207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.1:50067)
0620 09:02:24.288786 (+ 5579us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:24.288852 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:24.289961 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:24.293556 (+ 3595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:24.293565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:24.293904 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:24.294530 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.294547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.300088 (+ 5541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.300093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.301577 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.301591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.302951 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.302955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.303147 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.303812 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:24.303832 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:24.305453 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:24.317805 (+ 12352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.317812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.317826 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.318281 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.318762 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.318766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.318768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.318959 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.319710 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:24.319717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:24.320026 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 09:02:24.320214 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 09:02:24.323260 (+ 3046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5309,"thread_start_us":167,"threads_started":1}
I20260620 09:02:24.327561 13376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:24.327806 13376 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.328357 13376 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
W20260620 09:02:24.330575 13376 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38071 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:02:24.330708 13376 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.330878 13376 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:24.331869 13376 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.332037 13376 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
W20260620 09:02:24.332911 13376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:02:24.427198 13378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:24.427580 13378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:24.427651 13378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:24.433868 13378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:24.433948 13378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:24.434036 13378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:24.442062 13378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41809
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:24.443506 13378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:24.444875 13378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:24.455785 13385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:24.455873 13388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:24.455829 13386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:24.456305 13378 server_base.cc:1034] running on GCE node
I20260620 09:02:24.456745 13378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:24.457314 13378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:24.458501 13378 hybrid_clock.cc:648] HybridClock initialized: now 1781946144458469 us; error 44 us; skew 500 ppm
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:24.461294 13378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:24.462126 13378 webserver.cc:460] Webserver started at http://127.0.98.2:41809/ using document root <none> and password file <none>
I20260620 09:02:24.462656 13378 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:24.462759 13378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:24.487418 13378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 09:02:24.490008 13396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.491328 13378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:02:24.491460 13378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "0ce0de8e50bc4c5396c6fcd60e829371"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "be599cd542da7d49315dd59412230ef0"
server_key_iv: "338425aa891ead2e46c78d51e1a7b45e"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:24.491997 13378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:24.538666 13378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:24.542364 13378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:24.542563 13378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:24.543238 13378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:24.544253 13378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:24.544310 13378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.544365 13378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:24.544397 13378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.554414 13378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33899
I20260620 09:02:24.554555 13510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33899 every 8 connection(s)
I20260620 09:02:24.555600 13378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:02:24.559710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13378
I20260620 09:02:24.560181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38071
--webserver_port=41877
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:24.574383 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.558170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.2:48253)
0620 09:02:24.559435 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:24.559492 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:24.560983 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:24.561856 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:24.561865 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:24.562271 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:24.562946 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.562966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.564112 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.564116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.565519 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.565530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.566316 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.566320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.566524 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.567242 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:24.567263 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:24.568945 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:24.571045 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.571051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.571068 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.571425 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.571741 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.571744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.571746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.571922 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.572406 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:24.572413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:24.572783 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 09:02:24.572983 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 09:02:24.573334 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":142,"threads_started":1}
I20260620 09:02:24.575222 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.557592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48253 (local address 127.0.98.62:38071)
0620 09:02:24.557720 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:24.557725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:24.559616 (+ 1891us) server_negotiation.cc:407] Connection header received
0620 09:02:24.561294 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:24.561298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:24.561355 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:24.561410 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:24.563124 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.563956 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.565664 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.566023 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.566128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.569143 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:24.569147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:24.569149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:24.569179 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:24.570912 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.571551 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.571555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.571557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.571642 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.572026 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.572029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.572031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.572264 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:24.572353 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:24.574926 (+ 2573us) server_negotiation.cc:299] Negotiation successful
0620 09:02:24.575056 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:02:24.576333 13511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:24.576620 13511 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.577222 13511 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
W20260620 09:02:24.579063 13511 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38071 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:02:24.579200 13511 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.579385 13511 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:24.580287 13511 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.580443 13511 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
W20260620 09:02:24.581281 13511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:02:24.718618 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:24.718858 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:24.718899 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:24.723268 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:24.723352 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:24.723456 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:24.729545 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41877
--tserver_master_addrs=127.0.98.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:24.730649 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:24.731798 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:24.742710 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:24.743432 13514 server_base.cc:1034] running on GCE node
W20260620 09:02:24.743561 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:24.748929 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:24.749485 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:24.750432 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:24.751642 13514 hybrid_clock.cc:648] HybridClock initialized: now 1781946144751580 us; error 78 us; skew 500 ppm
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:24.755342 13514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:24.756482 13514 webserver.cc:460] Webserver started at http://127.0.98.3:41877/ using document root <none> and password file <none>
I20260620 09:02:24.757211 13514 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:24.757325 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:24.778038 13514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 09:02:24.782624 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.784420 13514 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260620 09:02:24.784623 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "bb64cf5e38d74c1884dede20f0633854"
format_stamp: "Formatted at 2026-06-20 09:02:23 on dist-test-slave-scsf"
server_key: "2ad706b7e447e02c6b436a8fa100afcb"
server_key_iv: "304eaa0bec881c7c0136c921d7d6c156"
server_key_version: "kuduclusterkey@0"
I20260620 09:02:24.785355 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:24.800549 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:24.803762 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:24.803963 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:24.804708 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:24.805759 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:24.805837 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.805905 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:24.805944 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:24.817498 13514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39569
I20260620 09:02:24.818871 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:24.819509 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39569 every 8 connection(s)
I20260620 09:02:24.825004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13514
Jun 20 09:02:24 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:24.836462 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.822864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57799 (local address 127.0.98.62:38071)
0620 09:02:24.822998 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:24.823002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:24.823021 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:02:24.823773 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:24.823776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:24.823828 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:24.823902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:24.826195 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.826992 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.828649 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.828996 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.829109 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.831999 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:24.832003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:24.832007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:24.832035 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:24.833906 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.834522 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.834525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.834527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.834624 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.835009 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.835013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.835015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.835228 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:24.835299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:24.836107 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 09:02:24.836257 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:02:24.838246 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.821720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.98.3:57799)
0620 09:02:24.822271 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:24.822326 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:24.823535 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:24.824078 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:24.824087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:24.824433 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:24.826027 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.826045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.827149 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.827153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.828524 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:24.828535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.829158 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.829180 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:24.829376 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.830071 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:24.830091 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:24.831817 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:24.834032 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.834038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.834054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.834399 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.834720 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:24.834723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:24.834725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:24.834896 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:24.835397 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:24.835403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:24.835743 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 09:02:24.836755 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0620 09:02:24.837148 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260620 09:02:24.839493 13644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38071
I20260620 09:02:24.839740 13644 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:24.840207 13644 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:24.841610 13178 ts_manager.cc:194] Registered new tserver with Master: bb64cf5e38d74c1884dede20f0633854 (127.0.98.3:39569)
I20260620 09:02:24.841837 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:24.826018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.98.62:38071)
0620 09:02:24.832642 (+ 6624us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:24.832652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:24.832668 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:02:24.832706 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:24.832711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:24.832764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:24.832864 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:24.834103 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.834728 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.835559 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:24.835780 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:24.835875 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:24.837505 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:24.837508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:24.837511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:24.837535 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:24.839193 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.840516 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.840520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.840521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.840601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:24.840919 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:24.840922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:24.840923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:24.841133 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:24.841314 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:24.841538 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 09:02:24.841653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6537,"thread_start_us":103,"threads_started":1}
I20260620 09:02:24.843060 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57799
Jun 20 09:02:25 dist-test-slave-scsf krb5kdc[12182](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:25.333149 13376 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:25.333460 13376 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:25.334247 13178 ts_manager.cc:194] Registered new tserver with Master: 349d47d17d314302ad30f0f7a5223e27 (127.0.98.1:35895)
I20260620 09:02:25.335171 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50067
Jun 20 09:02:25 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:02:25.581504 13511 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:25.581833 13511 heartbeater.cc:511] Master 127.0.98.62:38071 requested a full tablet report, sending...
I20260620 09:02:25.582640 13178 ts_manager.cc:194] Registered new tserver with Master: 0ce0de8e50bc4c5396c6fcd60e829371 (127.0.98.2:33899)
I20260620 09:02:25.583442 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48253
I20260620 09:02:25.592540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:02:25.604019 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.594115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.62:38071)
0620 09:02:25.594705 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.594710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.594723 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:02:25.594760 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.594762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.594808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.594905 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:25.595724 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.596662 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.597862 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.598234 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.598324 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.600048 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:25.600051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:25.600054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:25.600080 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:25.601975 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:25.602447 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:25.602450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:25.602452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:25.602534 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:25.602815 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:25.602818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:25.602820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:25.603026 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:25.603142 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.603737 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.603852 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":111,"threads_started":1}
I20260620 09:02:25.607605 13178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:02:25.612310 13178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:02:25.633556 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.625643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39569 (local address 127.0.0.1:57096)
0620 09:02:25.628580 (+ 2937us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.628593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:25.628695 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:25.629301 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:25.629304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:25.629320 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:25.629394 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.629399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.630707 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.630710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.631787 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.631797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.633025 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.633028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.633155 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.633193 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:02:25.633299 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:02:25.633408 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2878,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260620 09:02:25.634384 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.624708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.0.98.1:35895)
0620 09:02:25.625760 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.625767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.626133 (+ 366us) server_negotiation.cc:407] Connection header received
0620 09:02:25.626409 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.626415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.626617 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.626739 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:25.627833 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.629051 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.631850 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.632703 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.632795 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.632951 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.634119 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.634244 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":87,"threads_started":1}
I20260620 09:02:25.634698 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.625080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.0.98.2:33899)
0620 09:02:25.626421 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.626429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.626680 (+ 251us) server_negotiation.cc:407] Connection header received
0620 09:02:25.627158 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.627164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.627366 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.627483 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:25.627955 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.629138 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.633157 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.633936 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.634009 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.634159 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.634481 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.634578 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":72,"threads_started":1}
I20260620 09:02:25.634779 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.623865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35895 (local address 127.0.0.1:39174)
0620 09:02:25.626005 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.626053 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:25.626247 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:25.626943 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:25.626948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:25.626985 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:25.627103 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.627108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.629176 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.629180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.631680 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.631689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.633039 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.633042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.633958 (+ 916us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.633990 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 09:02:25.634574 (+ 584us) client_negotiation.cc:240] Negotiation successful
0620 09:02:25.634643 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1972,"mutex_wait_us":2099,"spinlock_wait_cycles":8960,"thread_start_us":101,"threads_started":1}
I20260620 09:02:25.635087 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.623913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33899 (local address 127.0.0.1:47954)
0620 09:02:25.626580 (+ 2667us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.626595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:25.626993 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:25.627672 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:25.627675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:25.627694 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:25.627820 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.627828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.629267 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.629272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.630572 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.630582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.634247 (+ 3665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.634250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.634332 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.634351 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:02:25.634891 (+ 540us) client_negotiation.cc:240] Negotiation successful
0620 09:02:25.634965 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2597,"spinlock_wait_cycles":3712,"thread_start_us":99,"threads_started":1}
I20260620 09:02:25.635725 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.626103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.0.98.3:39569)
0620 09:02:25.626900 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.626907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.628782 (+ 1875us) server_negotiation.cc:407] Connection header received
0620 09:02:25.628870 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.628874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.629034 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.629148 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:25.629513 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.630584 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.631906 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.632703 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.632803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.632951 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.635487 (+ 2536us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.635583 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":100,"threads_started":1}
I20260620 09:02:25.637872 13310 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9f55b71143482a8ade7aef4b065500 (DEFAULT_TABLE table=test-table [id=b0814ad6705f400392893a279648db2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:25.638581 13445 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9f55b71143482a8ade7aef4b065500 (DEFAULT_TABLE table=test-table [id=b0814ad6705f400392893a279648db2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:25.638578 13578 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9f55b71143482a8ade7aef4b065500 (DEFAULT_TABLE table=test-table [id=b0814ad6705f400392893a279648db2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:25.639071 13310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9f55b71143482a8ade7aef4b065500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:25.639544 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9f55b71143482a8ade7aef4b065500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:25.639544 13445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9f55b71143482a8ade7aef4b065500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:25.648674 13679 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Bootstrap starting.
I20260620 09:02:25.649477 13677 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Bootstrap starting.
I20260620 09:02:25.651010 13679 tablet_bootstrap.cc:654] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:25.651835 13677 tablet_bootstrap.cc:654] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:25.651904 13679 log.cc:824] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:25.652813 13677 log.cc:824] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:25.653966 13678 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Bootstrap starting.
I20260620 09:02:25.654233 13679 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: No bootstrap required, opened a new log
I20260620 09:02:25.654438 13679 ts_tablet_manager.cc:1397] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:02:25.656463 13678 tablet_bootstrap.cc:654] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:25.657269 13678 log.cc:824] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:25.658667 13679 raft_consensus.cc:348] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.658931 13679 raft_consensus.cc:374] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:25.658978 13679 raft_consensus.cc:729] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce0de8e50bc4c5396c6fcd60e829371, State: Initialized, Role: FOLLOWER
I20260620 09:02:25.659400 13677 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: No bootstrap required, opened a new log
I20260620 09:02:25.659564 13677 ts_tablet_manager.cc:1397] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 09:02:25.659487 13679 consensus_queue.cc:260] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.660465 13679 ts_tablet_manager.cc:1428] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:02:25.660840 13511 heartbeater.cc:503] Master 127.0.98.62:38071 was elected leader, sending a full tablet report...
I20260620 09:02:25.666895 13677 raft_consensus.cc:348] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.667186 13677 raft_consensus.cc:374] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:25.667251 13677 raft_consensus.cc:729] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb64cf5e38d74c1884dede20f0633854, State: Initialized, Role: FOLLOWER
I20260620 09:02:25.667708 13677 consensus_queue.cc:260] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.668431 13644 heartbeater.cc:503] Master 127.0.98.62:38071 was elected leader, sending a full tablet report...
I20260620 09:02:25.668766 13677 ts_tablet_manager.cc:1428] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20260620 09:02:25.670833 13678 tablet_bootstrap.cc:492] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: No bootstrap required, opened a new log
I20260620 09:02:25.671097 13678 ts_tablet_manager.cc:1397] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260620 09:02:25.675235 13678 raft_consensus.cc:348] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.675491 13678 raft_consensus.cc:374] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:25.675532 13678 raft_consensus.cc:729] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349d47d17d314302ad30f0f7a5223e27, State: Initialized, Role: FOLLOWER
I20260620 09:02:25.675954 13678 consensus_queue.cc:260] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.676614 13376 heartbeater.cc:503] Master 127.0.98.62:38071 was elected leader, sending a full tablet report...
I20260620 09:02:25.676944 13678 ts_tablet_manager.cc:1428] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:02:25.774933 13683 raft_consensus.cc:482] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:25.775175 13683 raft_consensus.cc:504] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.776814 13683 leader_election.cc:288] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349d47d17d314302ad30f0f7a5223e27 (127.0.98.1:35895), bb64cf5e38d74c1884dede20f0633854 (127.0.98.3:39569)
W20260620 09:02:25.782299 13377 tablet.cc:2369] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:02:25.784334 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.777367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59671 (local address 127.0.98.3:39569)
0620 09:02:25.777498 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.777504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.777771 (+ 267us) server_negotiation.cc:407] Connection header received
0620 09:02:25.778000 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.778003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.778060 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.778126 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:25.778675 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.779307 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.780859 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.781896 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.781985 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.782056 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.784102 (+ 2046us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.784189 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:02:25.785004 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.777236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39569 (local address 127.0.98.2:59671)
0620 09:02:25.777664 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.777679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:25.777859 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:25.778301 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:25.778304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:25.778321 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:25.778540 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.778547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.779424 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.779428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.780727 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.780736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.782154 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.782157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.782364 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.783971 (+ 1607us) client_negotiation.cc:769] Sending connection context
0620 09:02:25.784754 (+ 783us) client_negotiation.cc:240] Negotiation successful
0620 09:02:25.784838 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":1561,"spinlock_wait_cycles":22272,"thread_start_us":130,"threads_started":1}
I20260620 09:02:25.785876 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9f55b71143482a8ade7aef4b065500" candidate_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb64cf5e38d74c1884dede20f0633854" is_pre_election: true
I20260620 09:02:25.786255 13598 raft_consensus.cc:2454] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce0de8e50bc4c5396c6fcd60e829371 in term 0.
I20260620 09:02:25.786803 13399 leader_election.cc:302] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ce0de8e50bc4c5396c6fcd60e829371, bb64cf5e38d74c1884dede20f0633854; no voters:
I20260620 09:02:25.787120 13683 raft_consensus.cc:2781] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:02:25.787199 13683 raft_consensus.cc:482] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:02:25.787253 13683 raft_consensus.cc:3037] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:25.787266 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.777985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60335 (local address 127.0.98.1:35895)
0620 09:02:25.778091 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.778095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.778322 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:02:25.778523 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.778526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.778574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.778621 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:25.779651 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.780416 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.785952 (+ 5536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.786567 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.786649 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.786710 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.787052 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.787143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:02:25.787691 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.777638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35895 (local address 127.0.98.2:60335)
0620 09:02:25.778237 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.778252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:25.778405 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:25.779366 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:25.779372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:25.779397 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:25.779523 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.779528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.780579 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.780582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.785802 (+ 5220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:25.785813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.786811 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.786817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:25.786913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.786933 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:02:25.787452 (+ 519us) client_negotiation.cc:240] Negotiation successful
0620 09:02:25.787544 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260620 09:02:25.788281 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9f55b71143482a8ade7aef4b065500" candidate_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d47d17d314302ad30f0f7a5223e27" is_pre_election: true
I20260620 09:02:25.788599 13330 raft_consensus.cc:2454] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce0de8e50bc4c5396c6fcd60e829371 in term 0.
I20260620 09:02:25.788762 13683 raft_consensus.cc:504] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.789121 13683 leader_election.cc:288] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [CANDIDATE]: Term 1 election: Requested vote from peers 349d47d17d314302ad30f0f7a5223e27 (127.0.98.1:35895), bb64cf5e38d74c1884dede20f0633854 (127.0.98.3:39569)
I20260620 09:02:25.789549 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9f55b71143482a8ade7aef4b065500" candidate_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb64cf5e38d74c1884dede20f0633854"
I20260620 09:02:25.789568 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9f55b71143482a8ade7aef4b065500" candidate_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349d47d17d314302ad30f0f7a5223e27"
I20260620 09:02:25.789671 13598 raft_consensus.cc:3037] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:25.789671 13330 raft_consensus.cc:3037] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:25.790906 13330 raft_consensus.cc:2454] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce0de8e50bc4c5396c6fcd60e829371 in term 1.
I20260620 09:02:25.790906 13598 raft_consensus.cc:2454] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce0de8e50bc4c5396c6fcd60e829371 in term 1.
I20260620 09:02:25.791239 13397 leader_election.cc:302] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ce0de8e50bc4c5396c6fcd60e829371, 349d47d17d314302ad30f0f7a5223e27; no voters:
I20260620 09:02:25.791437 13683 raft_consensus.cc:2781] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:25.791680 13683 raft_consensus.cc:686] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce0de8e50bc4c5396c6fcd60e829371, State: Running, Role: LEADER
I20260620 09:02:25.792037 13683 consensus_queue.cc:237] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } }
I20260620 09:02:25.798524 13177 catalog_manager.cc:5495] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ce0de8e50bc4c5396c6fcd60e829371 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce0de8e50bc4c5396c6fcd60e829371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:02:25 dist-test-slave-scsf krb5kdc[12182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 09:02:25.812511 13512 tablet.cc:2369] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:02:25.820901 13645 tablet.cc:2369] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:02:25.825651 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.820270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.0.98.2:33899)
0620 09:02:25.820405 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:25.820410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:25.820423 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:02:25.820523 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:25.820526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:25.820589 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:25.820684 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:02:25.821713 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.822466 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.823629 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:25.823940 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:25.824031 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:25.824477 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:02:25.824951 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:02:25.825052 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:25.825453 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 09:02:25.825509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:02:25.834286 13598 raft_consensus.cc:1261] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 1 FOLLOWER]: Refusing update from remote peer 0ce0de8e50bc4c5396c6fcd60e829371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:25.834556 13330 raft_consensus.cc:1261] T 0c9f55b71143482a8ade7aef4b065500 P 349d47d17d314302ad30f0f7a5223e27 [term 1 FOLLOWER]: Refusing update from remote peer 0ce0de8e50bc4c5396c6fcd60e829371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:25.835170 13683 consensus_queue.cc:1035] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb64cf5e38d74c1884dede20f0633854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:25.835644 13689 consensus_queue.cc:1035] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349d47d17d314302ad30f0f7a5223e27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:25.859619 13178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:25.859786 13178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:25.862566 13178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c3a51d38e64a4aa0f00c169e2e5fb7: Sending DeleteTablet for 3 replicas of tablet 0c9f55b71143482a8ade7aef4b065500
I20260620 09:02:25.863927 13310 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9f55b71143482a8ade7aef4b065500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39174
I20260620 09:02:25.864435 13445 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9f55b71143482a8ade7aef4b065500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47954
I20260620 09:02:25.864898 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9f55b71143482a8ade7aef4b065500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57096
I20260620 09:02:25.866299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13216
I20260620 09:02:25.867012 13701 tablet_replica.cc:330] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: stopping tablet replica
I20260620 09:02:25.867309 13701 raft_consensus.cc:2229] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:02:25.867690 13701 raft_consensus.cc:2258] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:25.868059 13702 tablet_replica.cc:330] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: stopping tablet replica
I20260620 09:02:25.871213 13702 raft_consensus.cc:2229] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:25.871435 13701 ts_tablet_manager.cc:1905] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:25.871783 13702 raft_consensus.cc:2258] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:25.875973 13701 ts_tablet_manager.cc:1918] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:25.876086 13701 log.cc:1196] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/0c9f55b71143482a8ade7aef4b065500
I20260620 09:02:25.876420 13701 ts_tablet_manager.cc:1939] T 0c9f55b71143482a8ade7aef4b065500 P 0ce0de8e50bc4c5396c6fcd60e829371: Deleting consensus metadata
I20260620 09:02:25.879403 13164 catalog_manager.cc:4841] TS 0ce0de8e50bc4c5396c6fcd60e829371 (127.0.98.2:33899): tablet 0c9f55b71143482a8ade7aef4b065500 (table test-table [id=b0814ad6705f400392893a279648db2f]) successfully deleted
W20260620 09:02:25.883561 13162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35895 (error 108)
I20260620 09:02:25.885299 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:25.884925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35895 (local address 127.0.0.1:39190)
0620 09:02:25.885103 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:25.885172 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 09:02:25.885566 13162 catalog_manager.cc:4584] TS 349d47d17d314302ad30f0f7a5223e27 (127.0.98.1:35895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c9f55b71143482a8ade7aef4b065500: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35895: connect: Connection refused (error 111)
I20260620 09:02:25.885933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13378
I20260620 09:02:25.892174 13702 ts_tablet_manager.cc:1905] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:25.894979 13702 ts_tablet_manager.cc:1918] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:25.895072 13702 log.cc:1196] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/0c9f55b71143482a8ade7aef4b065500
I20260620 09:02:25.895386 13702 ts_tablet_manager.cc:1939] T 0c9f55b71143482a8ade7aef4b065500 P bb64cf5e38d74c1884dede20f0633854: Deleting consensus metadata
I20260620 09:02:25.895695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13514
I20260620 09:02:25.903318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13145
I20260620 09:02:25.916184 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:02:26.260710 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:02:26.260811 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:02:26.604919 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:02:26.605 UTC [12212] LOG: received smart shutdown request
2026-06-20 09:02:26.608 UTC [12212] LOG: background worker "logical replication launcher" (PID 12221) exited with exit code 1
2026-06-20 09:02:26.608 UTC [12216] LOG: shutting down
2026-06-20 09:02:26.747 UTC [12212] LOG: database system is shut down
2026-06-20T09:02:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46767 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:02:26 dist-test-slave-scsf krb5kdc[13714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:02:26 dist-test-slave-scsf krb5kdc[13714](info): set up 2 sockets
Jun 20 09:02:26 dist-test-slave-scsf krb5kdc[13714](info): commencing operation
krb5kdc: starting...
W20260620 09:02:27.884663 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:02:27 dist-test-slave-scsf krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:02:28.004221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:28.109567 13727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:28.109839 13727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:28.109905 13727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:28.113730 13727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:28.113806 13727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:28.113842 13727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:28.113911 13727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:28.113947 13727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:28.118687 13727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:28.119619 13727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:28.120512 13727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:28.128471 13735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:28.128437 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:28.128659 13727 server_base.cc:1034] running on GCE node
W20260620 09:02:28.128660 13733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:28.129271 13727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:28.130273 13727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:28.131451 13727 hybrid_clock.cc:648] HybridClock initialized: now 1781946148131439 us; error 32 us; skew 500 ppm
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:28.134028 13727 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:28.134778 13727 webserver.cc:460] Webserver started at http://127.0.98.62:46511/ using document root <none> and password file <none>
I20260620 09:02:28.135250 13727 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:28.135303 13727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:28.135447 13727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:28.137322 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "d1a418590f9c4383b9e6c8b56b9261f9"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "81628af2946e891250f20720634969a8"
server_key_iv: "9c227586aa4c82de1e9a586cea3eea4e"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.137796 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1a418590f9c4383b9e6c8b56b9261f9"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "81628af2946e891250f20720634969a8"
server_key_iv: "9c227586aa4c82de1e9a586cea3eea4e"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.141245 13727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:02:28.143369 13742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:28.144660 13727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:02:28.144752 13727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "d1a418590f9c4383b9e6c8b56b9261f9"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "81628af2946e891250f20720634969a8"
server_key_iv: "9c227586aa4c82de1e9a586cea3eea4e"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.144827 13727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:28.157258 13727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:28.159806 13727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:28.159999 13727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:28.166996 13727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43337
I20260620 09:02:28.167012 13794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43337 every 8 connection(s)
I20260620 09:02:28.167961 13727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:28.170164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13727
I20260620 09:02:28.170288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:02:28.170580 392 external_mini_cluster.cc:1428] Setting key ab48a0d8be44a3387ad82d0a49634382
I20260620 09:02:28.170915 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:28.174847 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:28.190222 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.177673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43337 (local address 127.0.0.1:40548)
0620 09:02:28.178147 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.178197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.179056 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.179706 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.179712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.180087 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.180322 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.180334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.181578 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.181581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.182282 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.182291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.182902 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.182906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.183054 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.183664 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.183679 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.185512 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.187310 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.187319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.187329 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.187656 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.187955 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.187960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.187962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.188118 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.188803 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.188809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.188968 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.189160 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.189495 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":105,"spinlock_wait_cycles":1792,"thread_start_us":132,"threads_started":1}
I20260620 09:02:28.190292 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.177673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.0.98.62:43337)
0620 09:02:28.178225 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.178241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.178293 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:02:28.179326 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.179331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.179498 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.179653 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.180484 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.181461 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.182402 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.182756 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.182975 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.185648 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.185653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.185663 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.185682 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.187188 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.187763 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.187767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.187768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.187862 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.188248 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.188252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.188253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.188683 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.188793 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.189229 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.189404 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":407,"thread_start_us":125,"threads_started":1}
I20260620 09:02:28.192423 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:02:28.193153 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.192926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59658)
0620 09:02:28.193044 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.193105 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:02:28.193356 13743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
I20260620 09:02:28.194135 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.193969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59662)
0620 09:02:28.194055 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.194094 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 09:02:28.194279 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33787: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:02:28.197186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33787
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:28.243593 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } attempt: 1
I20260620 09:02:28.244565 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.244211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59668)
0620 09:02:28.244376 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.244435 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 09:02:28.245240 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.245041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59678)
0620 09:02:28.245142 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.245192 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:02:28.245369 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33787: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
W20260620 09:02:28.302623 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:28.302904 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:28.302975 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:28.307050 13802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:28.307118 13802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:28.307153 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:28.307225 13802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:28.307263 13802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:28.312286 13802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:28.313284 13802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:28.313510 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } attempt: 2
I20260620 09:02:28.314244 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.313971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59690)
0620 09:02:28.314109 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.314171 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 09:02:28.314256 13802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:02:28.314901 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.314691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59694)
0620 09:02:28.314789 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.314830 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:02:28.315016 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33787: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33787: connect: Connection refused (error 111)
W20260620 09:02:28.323302 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:28.323304 13807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:28.323360 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:28.323505 13802 server_base.cc:1034] running on GCE node
I20260620 09:02:28.324173 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:28.325416 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:28.326597 13802 hybrid_clock.cc:648] HybridClock initialized: now 1781946148326582 us; error 25 us; skew 500 ppm
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:28.329169 13802 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:02:28.330097 13802 webserver.cc:460] Webserver started at http://127.0.98.61:44085/ using document root <none> and password file <none>
I20260620 09:02:28.330650 13802 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:28.330741 13802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:28.330967 13802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:28.332973 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data/instance:
uuid: "6ab63f8e19474daaae82415494a163d5"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "8842e2c0e6aa0ccb5b3ee87fa9b631d4"
server_key_iv: "34a14a9dc0c9f34658583dc9522ccc56"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.333521 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal/instance:
uuid: "6ab63f8e19474daaae82415494a163d5"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "8842e2c0e6aa0ccb5b3ee87fa9b631d4"
server_key_iv: "34a14a9dc0c9f34658583dc9522ccc56"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.337015 13802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:02:28.339241 13817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:28.340342 13802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:28.340512 13802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "6ab63f8e19474daaae82415494a163d5"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "8842e2c0e6aa0ccb5b3ee87fa9b631d4"
server_key_iv: "34a14a9dc0c9f34658583dc9522ccc56"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.340626 13802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:28.358953 13802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:28.361678 13802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:28.361862 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:28.369009 13802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33787
I20260620 09:02:28.369017 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33787 every 8 connection(s)
I20260620 09:02:28.369922 13802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:02:28.372663 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:28.372970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13802
I20260620 09:02:28.373131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-1/wal/instance
I20260620 09:02:28.373435 392 external_mini_cluster.cc:1428] Setting key a268c8eacc8026e17114c255839c1bfe
I20260620 09:02:28.376937 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:28.390852 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.378813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.98.62:43337)
0620 09:02:28.378940 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.378943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.379945 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 09:02:28.380926 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.380930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.380991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.381094 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.382181 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.382790 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.384119 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.384357 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.384489 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.387190 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.387193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.387196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.387224 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.388852 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.389333 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.389339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.389341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.389423 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.389791 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.389794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.389796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.389975 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.390073 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.390567 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.390687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:02:28.391801 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.379308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43337 (local address 127.0.0.1:40564)
0620 09:02:28.379801 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.379839 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.380729 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.381260 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.381267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.381585 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.382037 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.382049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.382909 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.382919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.384002 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.384012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.384551 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.384556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.384782 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.385492 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.385511 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.387030 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.388958 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.388965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.388979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.389232 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.389537 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.389540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.389542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.389678 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.390083 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.390091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.390336 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.390526 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.390839 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":144,"threads_started":1}
I20260620 09:02:28.393232 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:02:28.397333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41207
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:02:28.402151 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.393798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59696)
0620 09:02:28.393937 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.393947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.394001 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.394506 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.394509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.394665 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.394740 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.394744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.395623 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.395629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.396132 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.396139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.396883 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.396886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.397021 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.397555 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.397573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.398834 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.400718 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.400721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.400723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.400908 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.401172 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.401176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.401178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.401258 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.401656 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.401659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.401734 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.401840 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.401953 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":37}
I20260620 09:02:28.402211 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.393906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.0.98.61:33787)
0620 09:02:28.394194 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.394204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.394231 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:02:28.394261 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.394265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.394396 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.394527 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.394836 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.395520 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.396252 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.396771 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.396886 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.398953 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.398961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.398972 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.398992 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.400597 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.401004 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.401009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.401011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.401087 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.401342 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.401344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.401346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.401553 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.401678 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.401911 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.402021 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260620 09:02:28.404004 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:02:28.404738 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.404521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41190)
0620 09:02:28.404627 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.404685 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:02:28.404901 13818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
I20260620 09:02:28.405683 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.405500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41204)
0620 09:02:28.405592 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.405631 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:02:28.405821 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
I20260620 09:02:28.406153 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } attempt: 3
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:02:28.414559 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.406597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.98.61:33787)
0620 09:02:28.406691 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.406695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.406707 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:02:28.406802 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.406805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.406840 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.406919 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.407292 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.407766 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.408971 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.409643 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.409751 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.411784 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.411787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.411789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.411809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.413168 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.413620 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.413622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.413624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.413686 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.413935 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.413937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.413938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.414074 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.414141 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.414395 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.414470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:02:28.414703 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.406519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59704)
0620 09:02:28.406613 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.406629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.406715 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.406936 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.406938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.407094 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.407188 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.407192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.407865 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.407869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.408851 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.408860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.409754 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.409757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.409890 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.410335 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.410346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.411655 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.413295 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.413299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.413301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.413529 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.413769 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.413772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.413774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.413856 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.414153 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.414156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.414252 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.414352 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.414478 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 09:02:28.415371 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:02:28.416028 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.415756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41214)
0620 09:02:28.415890 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.415959 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 09:02:28.416710 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.416418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41216)
0620 09:02:28.416592 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.416642 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 09:02:28.416836 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
I20260620 09:02:28.447983 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 1
I20260620 09:02:28.448823 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.448508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41218)
0620 09:02:28.448657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.448732 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 09:02:28.449509 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.449298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41222)
0620 09:02:28.449406 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.449455 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:02:28.449632 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
I20260620 09:02:28.455139 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 1
I20260620 09:02:28.455796 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.455526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41224)
0620 09:02:28.455658 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.455714 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:02:28.456480 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.456238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41226)
0620 09:02:28.456354 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.456402 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:02:28.456629 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
W20260620 09:02:28.505837 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:28.506108 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:28.506179 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:28.509949 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:28.510030 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:28.510068 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:28.510103 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:28.510133 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:28.515631 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43337,127.0.98.61:33787,127.0.98.60:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:28.516669 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:28.517609 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:02:28.524783 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 2
I20260620 09:02:28.524887 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 2
W20260620 09:02:28.525430 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:28.525580 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.525317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41228)
0620 09:02:28.525422 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.525482 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 09:02:28.525557 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:28.525790 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:28.525676 13876 server_base.cc:1034] running on GCE node
I20260620 09:02:28.525897 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.525363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41238)
0620 09:02:28.525789 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.525833 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329}
I20260620 09:02:28.526676 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.526421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41252)
0620 09:02:28.526554 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.526609 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 09:02:28.526682 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.526456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41266)
0620 09:02:28.526579 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.526626 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 09:02:28.526794 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260620 09:02:28.526842 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
W20260620 09:02:28.526861 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41207: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41207: connect: Connection refused (error 111)
I20260620 09:02:28.527885 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:28.529098 13876 hybrid_clock.cc:648] HybridClock initialized: now 1781946148529079 us; error 26 us; skew 500 ppm
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:28.531708 13876 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:02:28.532684 13876 webserver.cc:460] Webserver started at http://127.0.98.60:33625/ using document root <none> and password file <none>
I20260620 09:02:28.533227 13876 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:28.533316 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:28.533537 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:28.535508 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data/instance:
uuid: "60da0d143d86422aae0ec5198276cd35"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "bfe2b0e3127630adccde6f066ab60603"
server_key_iv: "72a7ffdda51e3ade93c485fdde6b2909"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.536047 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal/instance:
uuid: "60da0d143d86422aae0ec5198276cd35"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "bfe2b0e3127630adccde6f066ab60603"
server_key_iv: "72a7ffdda51e3ade93c485fdde6b2909"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.539497 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:02:28.541766 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:28.542912 13876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:28.543043 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "60da0d143d86422aae0ec5198276cd35"
format_stamp: "Formatted at 2026-06-20 09:02:28 on dist-test-slave-scsf"
server_key: "bfe2b0e3127630adccde6f066ab60603"
server_key_iv: "72a7ffdda51e3ade93c485fdde6b2909"
server_key_version: "encryptionkey@0"
I20260620 09:02:28.543172 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:28.567022 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:28.569780 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:28.569973 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:28.576695 13876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41207
I20260620 09:02:28.576709 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41207 every 8 connection(s)
I20260620 09:02:28.577561 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:02:28.580101 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:28.583238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 13876
I20260620 09:02:28.583384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781945987698834-392-0/minicluster-data/master-2/wal/instance
I20260620 09:02:28.583624 392 external_mini_cluster.cc:1428] Setting key 95c89ac9385c1a87e6f4452c409c2c29
I20260620 09:02:28.583771 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:02:28.595194 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.585584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.62:43337)
0620 09:02:28.585702 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.585716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.585984 (+ 268us) server_negotiation.cc:407] Connection header received
0620 09:02:28.586358 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.586361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.586422 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.586506 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.587221 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.587703 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.588700 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.588974 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.589100 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.591583 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.591590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.591595 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.591620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.593077 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.593558 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.593563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.593565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.593640 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.594039 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.594042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.594043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.594275 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.594376 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.594886 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.595012 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:02:28.595902 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.585497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43337 (local address 127.0.0.1:40580)
0620 09:02:28.585853 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.585887 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.586174 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.586552 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.586557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.586870 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.587093 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.587105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.587813 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.587820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.588588 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.588596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.589139 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.589143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.589356 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.589908 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.589932 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.591449 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.593185 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.593195 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.593203 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.593449 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.593746 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.593749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.593751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.593939 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.594383 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.594390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.594662 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.594819 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.595126 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":26,"spinlock_wait_cycles":2944,"thread_start_us":108,"threads_started":1}
I20260620 09:02:28.595912 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.584896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.98.60:41207)
0620 09:02:28.585322 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.585340 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.585377 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:02:28.586181 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.586197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.586363 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.586536 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.587292 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.588127 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.589065 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.589440 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.589551 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.592111 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.592116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.592124 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.592144 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.593779 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.594206 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.594211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.594213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.594285 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.594544 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.594546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.594547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.594736 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.594922 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.595158 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.595292 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260620 09:02:28.597038 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:28.605813 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.597451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33787 (local address 127.0.0.1:59716)
0620 09:02:28.597609 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.597623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.597683 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.597897 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.597900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.598052 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.598190 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.598198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.599034 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.599040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.599842 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.599854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.600381 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.600383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.600494 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.600894 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.600907 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.602039 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.604389 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.604393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.604394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.604652 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.604920 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.604923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.604926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.605016 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.605355 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.605359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.605448 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.605546 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.605665 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 09:02:28.605859 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.597566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.0.98.61:33787)
0620 09:02:28.597677 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.597681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.597693 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:02:28.597762 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.597763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.597796 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.597886 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.598298 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.598933 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.599951 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.600280 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.600379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.602182 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.602187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.602189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.602236 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.604261 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.604757 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.604760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.604761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.604831 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.605103 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.605105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.605106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.605259 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.605365 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.605606 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.605724 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:02:28.606583 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:02:28.607012 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.598302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.0.98.62:43337)
0620 09:02:28.598420 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.598423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.598435 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:02:28.598511 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.598514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.598551 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.598618 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.599096 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.599806 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.600772 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.600998 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.601112 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.603563 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.603567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.603569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.603597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.605114 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.605462 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.605467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.605468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.605545 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.605805 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.605809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.605811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.606384 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.606592 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.606747 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.606852 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:02:28.615157 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.607033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41298)
0620 09:02:28.607228 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.607240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.607315 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.607710 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.607712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.607826 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.607887 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.607891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.608660 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.608666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.609359 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.609367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.609917 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.609921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.609995 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.610349 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.610362 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.611779 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.613661 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.613664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.613666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.613911 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.614212 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.614214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.614215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.614315 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.614641 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.614644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.614735 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.614850 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.614976 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 09:02:28.615378 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.607072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.98.60:41207)
0620 09:02:28.607214 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.607217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.607540 (+ 323us) server_negotiation.cc:407] Connection header received
0620 09:02:28.607580 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.607582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.607622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.607688 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.608017 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.608545 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.609458 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.609802 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.609887 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.611900 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.611903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.611904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.611922 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.613564 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.614022 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.614024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.614025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.614089 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.614405 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.614407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.614408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.614548 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.614617 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.614890 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.615023 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":66}
I20260620 09:02:28.616595 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.608086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.0.98.61:33787)
0620 09:02:28.608180 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.608183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.608262 (+ 79us) server_negotiation.cc:407] Connection header received
0620 09:02:28.608389 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.608392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.608426 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.608516 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.609136 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.609689 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.610656 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.610844 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.610930 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.613444 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.613447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.613448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.613467 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.614989 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.615399 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.615405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.615408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.615498 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.615776 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.615779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.615780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.615960 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.616217 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.616288 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.616390 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:02:28.617992 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 3
I20260620 09:02:28.619022 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: Bootstrap starting.
I20260620 09:02:28.619822 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:02:28.621870 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:28.623560 13945 log.cc:824] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:28.625677 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: No bootstrap required, opened a new log
I20260620 09:02:28.626204 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } attempt: 3
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:02:28.630225 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.630502 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:28.630574 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da0d143d86422aae0ec5198276cd35, State: Initialized, Role: FOLLOWER
I20260620 09:02:28.631146 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.621984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.0.98.61:33787)
0620 09:02:28.622405 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.622411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.622477 (+ 66us) server_negotiation.cc:407] Connection header received
0620 09:02:28.622650 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.622653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.622690 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.622760 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.623596 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.624260 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.625321 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.625601 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.626832 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.627443 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.627447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.627451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.627476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.629084 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.629551 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.629558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.629560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.629656 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.629944 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.629947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.629949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.630149 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.630316 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.630903 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.631001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1149,"server-negotiator.queue_time_us":355}
I20260620 09:02:28.631908 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.621944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.62:43337)
0620 09:02:28.622057 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.622061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.622832 (+ 771us) server_negotiation.cc:407] Connection header received
0620 09:02:28.623694 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.623697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.623740 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.623785 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.624656 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.625357 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.626409 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.626691 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.626752 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.628318 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.628323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.628328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.628355 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.629956 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.630476 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.630481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.630483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.630572 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.630910 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.630915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.630917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.631159 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.631278 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.631632 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.631741 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:02:28.632495 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.633518 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.618388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41304)
0620 09:02:28.622100 (+ 3712us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.622113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.622232 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.622857 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.622861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.623044 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.623137 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.623143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.624095 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.624099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.624953 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.624962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.625561 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.625564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.625641 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.627097 (+ 1456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.627113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.629907 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.631847 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.631850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.631852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.632099 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.632403 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.632409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.632414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.632526 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.633012 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.633017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.633110 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.633206 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.633351 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3635,"mutex_wait_us":3}
I20260620 09:02:28.634037 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.618420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.0.98.60:41207)
0620 09:02:28.618546 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.618551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.622193 (+ 3642us) server_negotiation.cc:407] Connection header received
0620 09:02:28.622496 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.622499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.622538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.622637 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.623276 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.623970 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.625069 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.625362 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.625450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.630068 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.630071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.630073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.630095 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.631740 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.632209 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.632212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.632214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.632295 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.632636 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.632642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.632653 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.632876 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.632967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.633795 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.633899 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:02:28.634130 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:28.637019 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.637234 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.637290 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: Bootstrap starting.
Jun 20 09:02:28 dist-test-slave-scsf krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:02:28.640635 13870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:28.637511 13876 master_runner.cc:186] Error getting master registration for 127.0.98.62:43337: OK, instance_id { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" instance_seqno: 1781946148165068 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:02:28.641326 13876 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:02:28.641901 13870 log.cc:824] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:28.642937 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.626530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41207 (local address 127.0.0.1:41326)
0620 09:02:28.626678 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:28.626690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:28.626753 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:28.631399 (+ 4646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:28.631402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:28.631568 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:28.631653 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.631659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.634789 (+ 3130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.634792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.635660 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:28.635674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.636302 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.636308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:28.636436 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.637018 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:28.637035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:28.638463 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:28.641329 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.641331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.641335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.641550 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.641808 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:28.641810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:28.641812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:28.641906 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:28.642386 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:28.642389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:28.642485 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:02:28.642595 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 09:02:28.642744 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 09:02:28.643354 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.626586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.0.98.60:41207)
0620 09:02:28.627875 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.627883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.627896 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:02:28.628240 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.628243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.628287 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.631759 (+ 3472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.633795 (+ 2036us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x3f3f9028. stack: 00007f1c0a13ac78 00007f1c0a13b1a9 00007f1c09cfd1d7 00000000005984a6 00000000005984f9 000000000059c488 00007f1c0a1372ab 00007f1c0a1369da 00007f1c12894817 00007f1c128a707f 00007f1c0e820c99 00007f1c0e815e9d 00007f1c0e7cf219 00007f1c0e7cf615 00007f1c0e7e50d3 00007f1c0e7e854b
0620 09:02:28.633894 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.634680 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.635809 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.636162 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.636655 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.638595 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.638598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.638601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.638623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.641196 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.641643 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.641645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.641646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.641705 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.641990 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.641995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.641998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.642250 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.642348 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.643136 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.643237 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"spinlock_wait_cycles":3629440,"thread_start_us":80,"threads_started":1}
I20260620 09:02:28.644747 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: No bootstrap required, opened a new log
I20260620 09:02:28.645228 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:28.622344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.98.60:41207)
0620 09:02:28.622863 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:28.622873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:28.628755 (+ 5882us) server_negotiation.cc:407] Connection header received
0620 09:02:28.629268 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:28.629272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:28.633167 (+ 3895us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:28.633260 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:28.635556 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.636285 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.637878 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:28.638736 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:28.638832 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:28.640561 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:28.640568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:28.640570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:28.640596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:28.643599 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.644061 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.644064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.644065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.644154 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:28.644393 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:28.644397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:28.644399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:28.644611 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:28.644733 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:28.644972 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 09:02:28.645085 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":92,"threads_started":1}
I20260620 09:02:28.648028 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.648250 13870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:28.648307 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab63f8e19474daaae82415494a163d5, State: Initialized, Role: FOLLOWER
I20260620 09:02:28.649050 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.649489 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.649614 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.650313 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:28.650990 13795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: Bootstrap starting.
W20260620 09:02:28.654784 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:02:28.654794 13971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:02:28.654855 13971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 09:02:28.654896 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:02:28.656095 13795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:28.653339 13802 master_runner.cc:186] Error getting master registration for 127.0.98.62:43337: OK, instance_id { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" instance_seqno: 1781946148165068 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:02:28.656960 13795 log.cc:824] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:28.656992 13802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:02:28.658921 13795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: No bootstrap required, opened a new log
I20260620 09:02:28.660830 13973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:28.660984 13973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.661909 13973 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a418590f9c4383b9e6c8b56b9261f9 (127.0.98.62:43337), 60da0d143d86422aae0ec5198276cd35 (127.0.98.60:41207)
I20260620 09:02:28.662905 13795 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.663043 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab63f8e19474daaae82415494a163d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da0d143d86422aae0ec5198276cd35" is_pre_election: true
I20260620 09:02:28.663134 13795 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:28.663190 13795 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a418590f9c4383b9e6c8b56b9261f9, State: Initialized, Role: FOLLOWER
I20260620 09:02:28.663331 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab63f8e19474daaae82415494a163d5 in term 0.
I20260620 09:02:28.663622 13795 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.663756 13818 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60da0d143d86422aae0ec5198276cd35, 6ab63f8e19474daaae82415494a163d5; no voters:
I20260620 09:02:28.664033 13973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:02:28.664111 13973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:02:28.664160 13973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:28.664386 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.664734 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.664965 13795 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:28.665005 13769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab63f8e19474daaae82415494a163d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" is_pre_election: true
I20260620 09:02:28.665247 13769 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab63f8e19474daaae82415494a163d5 in term 0.
I20260620 09:02:28.665416 13973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.665726 13973 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [CANDIDATE]: Term 1 election: Requested vote from peers d1a418590f9c4383b9e6c8b56b9261f9 (127.0.98.62:43337), 60da0d143d86422aae0ec5198276cd35 (127.0.98.60:41207)
I20260620 09:02:28.666039 13769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab63f8e19474daaae82415494a163d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a418590f9c4383b9e6c8b56b9261f9"
I20260620 09:02:28.666039 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab63f8e19474daaae82415494a163d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da0d143d86422aae0ec5198276cd35"
I20260620 09:02:28.666139 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:28.666177 13769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:28.667260 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab63f8e19474daaae82415494a163d5 in term 1.
I20260620 09:02:28.667260 13769 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab63f8e19474daaae82415494a163d5 in term 1.
I20260620 09:02:28.667551 13819 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60da0d143d86422aae0ec5198276cd35, 6ab63f8e19474daaae82415494a163d5, d1a418590f9c4383b9e6c8b56b9261f9; no voters:
I20260620 09:02:28.667716 13973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:28.667927 13973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab63f8e19474daaae82415494a163d5, State: Running, Role: LEADER
I20260620 09:02:28.668272 13973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } }
I20260620 09:02:28.670747 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab63f8e19474daaae82415494a163d5. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.670853 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:28.672897 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:28.675064 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:28.680060 13919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [term 1 FOLLOWER]: Refusing update from remote peer 6ab63f8e19474daaae82415494a163d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:28.680934 13769 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [term 1 FOLLOWER]: Refusing update from remote peer 6ab63f8e19474daaae82415494a163d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 09:02:28.681419 14000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:02:28.681524 14000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:02:28.681800 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:28.682209 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:28.685555 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab63f8e19474daaae82415494a163d5. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.685680 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.688905 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab63f8e19474daaae82415494a163d5. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.688995 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.696889 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.697525 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.697922 14003 mvcc.cc:205] Tried to move back new op lower bound from 7298851424985178112 to 7298851424949084160. Current Snapshot: MvccSnapshot[applied={T|T < 7298851424985178112}]
I20260620 09:02:28.697347 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.698036 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:28.699361 14007 mvcc.cc:205] Tried to move back new op lower bound from 7298851424985178112 to 7298851424949084160. Current Snapshot: MvccSnapshot[applied={T|T < 7298851424985178112}]
I20260620 09:02:28.701478 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.701578 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:28.702914 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ab63f8e19474daaae82415494a163d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a418590f9c4383b9e6c8b56b9261f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43337 } } peers { permanent_uuid: "6ab63f8e19474daaae82415494a163d5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33787 } } peers { permanent_uuid: "60da0d143d86422aae0ec5198276cd35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41207 } } }
I20260620 09:02:28.703071 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:02:28.704335 14005 mvcc.cc:205] Tried to move back new op lower bound from 7298851424985178112 to 7298851424949084160. Current Snapshot: MvccSnapshot[applied={T|T < 7298851424985178112}]
I20260620 09:02:28.706563 14002 catalog_manager.cc:1347] Generated new cluster ID: 4b0f49acc8fd4446a0d607cc15e56c94
I20260620 09:02:28.706694 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:28.726701 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:02:28.727897 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:28.739710 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab63f8e19474daaae82415494a163d5: Generated new TSK 0
I20260620 09:02:28.740650 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:02:29.656102 13971 catalog_manager.cc:1259] Loaded cluster ID: 4b0f49acc8fd4446a0d607cc15e56c94
I20260620 09:02:29.656219 13971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: loading cluster ID for follower catalog manager: success
I20260620 09:02:29.657917 13971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: acquiring CA information for follower catalog manager: success
I20260620 09:02:29.659096 13971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60da0d143d86422aae0ec5198276cd35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:02:29.682781 14000 catalog_manager.cc:1259] Loaded cluster ID: 4b0f49acc8fd4446a0d607cc15e56c94
I20260620 09:02:29.682862 14000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: loading cluster ID for follower catalog manager: success
I20260620 09:02:29.684540 14000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: acquiring CA information for follower catalog manager: success
I20260620 09:02:29.685726 14000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1a418590f9c4383b9e6c8b56b9261f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:02:29.823313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13727
I20260620 09:02:29.829396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13802
I20260620 09:02:29.835080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 13876
2026-06-20T09:02:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2967 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:02:29 dist-test-slave-scsf krb5kdc[14019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:02:29 dist-test-slave-scsf krb5kdc[14019](info): set up 2 sockets
Jun 20 09:02:29 dist-test-slave-scsf krb5kdc[14019](info): commencing operation
krb5kdc: starting...
W20260620 09:02:30.860668 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:02:30 dist-test-slave-scsf krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:02:30.975689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:31.080621 14032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:31.080932 14032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:31.081004 14032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:31.084913 14032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:31.085001 14032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:31.085038 14032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:31.085093 14032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:31.085130 14032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:31.090121 14032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:31.091128 14032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:31.092114 14032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:31.100620 14038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:31.100620 14037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:31.100769 14032 server_base.cc:1034] running on GCE node
W20260620 09:02:31.101022 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:31.101536 14032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:31.102578 14032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:31.103760 14032 hybrid_clock.cc:648] HybridClock initialized: now 1781946151103733 us; error 43 us; skew 500 ppm
Jun 20 09:02:31 dist-test-slave-scsf krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:31.106348 14032 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:31.107215 14032 webserver.cc:460] Webserver started at http://127.0.98.62:34649/ using document root <none> and password file <none>
I20260620 09:02:31.107758 14032 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:31.107846 14032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:31.108064 14032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:31.110044 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "09e937993cc242dea316874eaa3cb9f9"
format_stamp: "Formatted at 2026-06-20 09:02:31 on dist-test-slave-scsf"
server_key: "4f3e8d6e278f6c3c39056189a133f457"
server_key_iv: "58c458d59ab97765e6b95fd8297a863e"
server_key_version: "encryptionkey@0"
I20260620 09:02:31.110574 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "09e937993cc242dea316874eaa3cb9f9"
format_stamp: "Formatted at 2026-06-20 09:02:31 on dist-test-slave-scsf"
server_key: "4f3e8d6e278f6c3c39056189a133f457"
server_key_iv: "58c458d59ab97765e6b95fd8297a863e"
server_key_version: "encryptionkey@0"
I20260620 09:02:31.114048 14032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 09:02:31.116217 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:31.117673 14032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:31.117833 14032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "09e937993cc242dea316874eaa3cb9f9"
format_stamp: "Formatted at 2026-06-20 09:02:31 on dist-test-slave-scsf"
server_key: "4f3e8d6e278f6c3c39056189a133f457"
server_key_iv: "58c458d59ab97765e6b95fd8297a863e"
server_key_version: "encryptionkey@0"
I20260620 09:02:31.117949 14032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:31.130900 14032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:31.133548 14032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:31.133747 14032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:31.140228 14032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46285
I20260620 09:02:31.140241 14099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46285 every 8 connection(s)
I20260620 09:02:31.141101 14032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:31.143628 14100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:31.148725 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: Bootstrap starting.
I20260620 09:02:31.150985 14100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:31.151171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14032
I20260620 09:02:31.151314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:02:31.151589 392 external_mini_cluster.cc:1428] Setting key 6514a7440da54616132f4ba38b19de7d
I20260620 09:02:31.151813 14100 log.cc:824] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:31.154023 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: No bootstrap required, opened a new log
I20260620 09:02:31.158303 14100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 09:02:31.158566 14100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:31.158674 14100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e937993cc242dea316874eaa3cb9f9, State: Initialized, Role: FOLLOWER
I20260620 09:02:31.159210 14100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 09:02:31.159381 14100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:31.159456 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:31.159552 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:31.160385 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
Jun 20 09:02:31 dist-test-slave-scsf krb5kdc[14019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:31.160753 14100 leader_election.cc:302] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e937993cc242dea316874eaa3cb9f9; no voters:
I20260620 09:02:31.161072 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:02:31.161198 14105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:31.161424 14105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 09e937993cc242dea316874eaa3cb9f9, State: Running, Role: LEADER
I20260620 09:02:31.161788 14105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 09:02:31.162357 14100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:31.163524 14106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e937993cc242dea316874eaa3cb9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260620 09:02:31.163645 14106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:31.164031 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e937993cc242dea316874eaa3cb9f9. Latest consensus state: current_term: 1 leader_uuid: "09e937993cc242dea316874eaa3cb9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e937993cc242dea316874eaa3cb9f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260620 09:02:31.164211 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:31.166604 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:31.152661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.0.98.62:46285)
0620 09:02:31.153075 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:31.153094 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:31.153136 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:02:31.153980 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:31.154000 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:31.154390 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:31.154741 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:31.155381 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:31.156889 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:31.157707 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:31.158410 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:31.158563 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:31.161313 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:31.161321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:31.161336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:31.161366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:31.163479 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:31.164024 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:31.164031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:31.164037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:31.164183 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:31.164534 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:31.164537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:31.164539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:31.164994 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:31.165188 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:31.165453 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 09:02:31.165754 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
W20260620 09:02:31.167294 14120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:02:31.167366 14120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:02:31.167477 14114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:31.168359 14114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:31.169448 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:02:31.173883 14114 catalog_manager.cc:1347] Generated new cluster ID: b45c1aa1b8994feaa6eb8c9486652932
I20260620 09:02:31.173945 14114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:31.189560 14114 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:02:31.190454 14114 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:31.194069 14114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09e937993cc242dea316874eaa3cb9f9: Generated new TSK 0
I20260620 09:02:31.194762 14114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:02:31.225257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14032
2026-06-20T09:02:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1388 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:02:31 dist-test-slave-scsf krb5kdc[14127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:02:31 dist-test-slave-scsf krb5kdc[14127](info): set up 2 sockets
Jun 20 09:02:31 dist-test-slave-scsf krb5kdc[14127](info): commencing operation
krb5kdc: starting...
W20260620 09:02:32.264745 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:02:32.383792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:32.490739 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:32.491052 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:32.491123 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:32.495195 14140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:32.495286 14140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:32.495326 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:32.495394 14140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:32.495433 14140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:32.500751 14140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:32.501796 14140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:32.502815 14140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:02:32.511257 14140 server_base.cc:1034] running on GCE node
W20260620 09:02:32.511205 14148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.511220 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.511184 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:32.511992 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:32.513124 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:32.514283 14140 hybrid_clock.cc:648] HybridClock initialized: now 1781946152514253 us; error 47 us; skew 500 ppm
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:32.521201 14140 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:32.522181 14140 webserver.cc:460] Webserver started at http://127.0.98.62:43053/ using document root <none> and password file <none>
I20260620 09:02:32.522770 14140 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:32.522863 14140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:32.523100 14140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:32.525316 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "f7c9fb88d4ce46f0955d89d611dec978"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "9860d52649773cfffab44efcdac9a3ad"
server_key_iv: "47d01d44248d600e6daed06cacf4ce79"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.525903 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7c9fb88d4ce46f0955d89d611dec978"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "9860d52649773cfffab44efcdac9a3ad"
server_key_iv: "47d01d44248d600e6daed06cacf4ce79"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.529728 14140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:02:32.532119 14155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:32.533460 14140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:32.533607 14140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "f7c9fb88d4ce46f0955d89d611dec978"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "9860d52649773cfffab44efcdac9a3ad"
server_key_iv: "47d01d44248d600e6daed06cacf4ce79"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.533722 14140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:32.555876 14140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:32.558828 14140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:32.559036 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:32.566347 14140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45337
I20260620 09:02:32.566607 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45337 every 8 connection(s)
I20260620 09:02:32.567348 14140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:32.569785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14140
I20260620 09:02:32.569952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:02:32.570186 14208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:32.570268 392 external_mini_cluster.cc:1428] Setting key b24aff0c635d16d5d09e64d6f0e38987
I20260620 09:02:32.577358 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap starting.
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:32.579694 14208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:32.580554 14208 log.cc:824] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:32.582820 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: No bootstrap required, opened a new log
I20260620 09:02:32.585343 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:32.571679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.62:45337)
0620 09:02:32.572095 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:32.572114 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:32.572160 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:02:32.572967 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:32.572991 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:32.573411 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:32.573777 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:32.574433 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.575918 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.576836 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.577512 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.577667 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:32.580293 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:32.580299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:32.580316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:32.580349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:32.582330 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:32.582752 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:32.582757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:32.582762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:32.582901 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:32.583175 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:32.583179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:32.583181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:32.583658 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:32.583849 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:32.584028 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:02:32.584324 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260620 09:02:32.587111 14208 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:32.587384 14208 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:32.587471 14208 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Initialized, Role: FOLLOWER
I20260620 09:02:32.587904 14208 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:32.588024 14208 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:32.588105 14208 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:32.588219 14208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:32.589296 14208 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:32.589640 14208 leader_election.cc:302] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c9fb88d4ce46f0955d89d611dec978; no voters:
I20260620 09:02:32.589958 14208 leader_election.cc:288] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:02:32.590073 14213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:32.590322 14213 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 LEADER]: Becoming Leader. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Running, Role: LEADER
I20260620 09:02:32.590723 14213 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:32.591113 14208 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:32.592406 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c9fb88d4ce46f0955d89d611dec978. Latest consensus state: current_term: 1 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:32.592383 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:32.592553 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:32.592553 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:32.593011 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:32.595784 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:32.601038 14224 catalog_manager.cc:1347] Generated new cluster ID: 71a80ba93aeb48ff8ed6ac6fd8fa5e2d
I20260620 09:02:32.601092 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:32.612692 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:02:32.613909 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:32.620003 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Generated new TSK 0
I20260620 09:02:32.620733 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:02:32.649703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:32.760937 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:32.761200 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:32.761243 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:32.765107 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:32.765162 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:32.765218 14234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:32.770246 14234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:32.771188 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:32.772110 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:32.781376 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.781394 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.781450 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:32.781589 14234 server_base.cc:1034] running on GCE node
I20260620 09:02:32.782075 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:32.782696 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:32.783910 14234 hybrid_clock.cc:648] HybridClock initialized: now 1781946152783874 us; error 48 us; skew 500 ppm
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:32.786772 14234 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:32.787601 14234 webserver.cc:460] Webserver started at http://127.0.98.1:43437/ using document root <none> and password file <none>
I20260620 09:02:32.788203 14234 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:32.788295 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:32.788578 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:32.790544 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "fee373f99902138875f26973c5639ecf"
server_key_iv: "77a4a82fdd8b26003f737f46a42611bf"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.791131 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "fee373f99902138875f26973c5639ecf"
server_key_iv: "77a4a82fdd8b26003f737f46a42611bf"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.795361 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:02:32.797847 14249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:32.799064 14234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:32.799197 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "fee373f99902138875f26973c5639ecf"
server_key_iv: "77a4a82fdd8b26003f737f46a42611bf"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.799305 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:32.813701 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:32.816836 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:32.817021 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:32.817572 14234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:32.818380 14234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:32.818454 14234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:32.818523 14234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:32.818565 14234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:32.827225 14234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46765
I20260620 09:02:32.827268 14362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46765 every 8 connection(s)
I20260620 09:02:32.828060 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:32.835508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14234
I20260620 09:02:32.835656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:02:32.835945 392 external_mini_cluster.cc:1428] Setting key d4c959d3b32839a25fd84359ef49b4e5
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:32.842262 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:32.829767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47145 (local address 127.0.98.62:45337)
0620 09:02:32.829895 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:32.829899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:32.830705 (+ 806us) server_negotiation.cc:407] Connection header received
0620 09:02:32.832003 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:32.832007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:32.832064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:32.832144 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:32.833331 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.833878 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.835029 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.835314 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.835422 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:32.838299 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:32.838305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:32.838311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:32.838344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:32.839889 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:32.840561 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:32.840565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:32.840566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:32.840665 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:32.841092 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:32.841094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:32.841095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:32.841301 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:32.841406 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:32.842046 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 09:02:32.842140 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:02:32.843508 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:32.830121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:47145)
0620 09:02:32.830547 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:32.830585 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:32.831797 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:32.832307 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:32.832314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:32.832728 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:32.833185 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:32.833197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.834007 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.834011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:32.834898 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:32.834908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:32.835471 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:32.835475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:32.835703 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:32.836352 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:32.836371 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:32.838138 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:32.840015 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:32.840024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:32.840040 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:32.840418 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:32.840782 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:32.840788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:32.840793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:32.840990 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:32.841434 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:32.841444 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:32.841783 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 09:02:32.841997 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 09:02:32.842360 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":123,"threads_started":1}
I20260620 09:02:32.844790 14363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:32.845032 14363 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:32.845484 14363 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:02:32.846937 14172 ts_manager.cc:194] Registered new tserver with Master: 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:32.848086 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:02:32.863111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:32.968183 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:32.968461 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:32.968564 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:32.972517 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:32.972566 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:32.972671 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:32.977713 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:32.978711 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:32.979782 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:32.988791 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.988807 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:32.988795 14376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:32.988873 14368 server_base.cc:1034] running on GCE node
I20260620 09:02:32.989358 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:32.989933 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:32.991098 14368 hybrid_clock.cc:648] HybridClock initialized: now 1781946152991064 us; error 53 us; skew 500 ppm
Jun 20 09:02:32 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:32.993971 14368 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:32.994750 14368 webserver.cc:460] Webserver started at http://127.0.98.2:33873/ using document root <none> and password file <none>
I20260620 09:02:32.995220 14368 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:32.995285 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:32.995434 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:32.997365 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "52bff167cf4642d9b10d3c63d3c884c1"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "3ecab00b3ed1bab11832fb476128a5d5"
server_key_iv: "e97e0cb24d653cb105f4b6677cfb7ec5"
server_key_version: "encryptionkey@0"
I20260620 09:02:32.997865 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52bff167cf4642d9b10d3c63d3c884c1"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "3ecab00b3ed1bab11832fb476128a5d5"
server_key_iv: "e97e0cb24d653cb105f4b6677cfb7ec5"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.001462 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:02:33.003705 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.004956 14368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:33.005072 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "52bff167cf4642d9b10d3c63d3c884c1"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "3ecab00b3ed1bab11832fb476128a5d5"
server_key_iv: "e97e0cb24d653cb105f4b6677cfb7ec5"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.005169 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:33.022887 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:33.025622 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:33.025818 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:33.026384 14368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:33.027171 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:33.027216 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.027290 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:33.027331 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.035974 14368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260620 09:02:33.036001 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260620 09:02:33.036873 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:02:33.038661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14368
I20260620 09:02:33.038792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:02:33.039064 392 external_mini_cluster.cc:1428] Setting key 14e09a2114fb909b3218d16d4b028fff
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:02:33.052284 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.038681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54737 (local address 127.0.98.62:45337)
0620 09:02:33.038804 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.038808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.039822 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 09:02:33.041451 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.041454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.041495 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.041561 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:33.042933 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.043667 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.045509 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.045797 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.045877 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.048567 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:33.048570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:33.048572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:33.048594 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:33.050157 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.050659 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.050667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.050669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.050747 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.051154 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.051157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.051159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.051340 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:33.051448 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.051992 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.052113 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:02:33.053061 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.039189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:54737)
0620 09:02:33.039695 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.039744 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.041237 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.041760 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.041771 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.042273 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:33.042793 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.042806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.043796 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.043803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.045386 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.045401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.045950 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.045957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.046190 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.046781 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:33.046801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:33.048352 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:33.050278 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.050284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.050298 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.050546 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.050847 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.050850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.050852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.051048 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.051489 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:33.051496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:33.051759 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.051936 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.052250 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260620 09:02:33.054200 14497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:33.054432 14497 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:33.054999 14497 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
I20260620 09:02:33.056054 14172 ts_manager.cc:194] Registered new tserver with Master: 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
I20260620 09:02:33.056680 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:02:33.067126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:33.174230 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:33.174481 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:33.174521 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:33.178522 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:33.178571 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:33.178674 14502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:33.183702 14502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:33.184732 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:33.185751 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:33.195184 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.195212 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:33.195282 14502 server_base.cc:1034] running on GCE node
W20260620 09:02:33.195184 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:33.195883 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:33.196578 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:33.197758 14502 hybrid_clock.cc:648] HybridClock initialized: now 1781946153197728 us; error 52 us; skew 500 ppm
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:33.200505 14502 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:33.201356 14502 webserver.cc:460] Webserver started at http://127.0.98.3:38741/ using document root <none> and password file <none>
I20260620 09:02:33.201898 14502 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:33.201987 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:33.202216 14502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:33.204237 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "ae364666861848eeb06c503416b6a4ca"
format_stamp: "Formatted at 2026-06-20 09:02:33 on dist-test-slave-scsf"
server_key: "f2b675d8d6acbd666ec878498b0690a3"
server_key_iv: "7fbb8be65de0788208e37f2bd9140462"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.204815 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ae364666861848eeb06c503416b6a4ca"
format_stamp: "Formatted at 2026-06-20 09:02:33 on dist-test-slave-scsf"
server_key: "f2b675d8d6acbd666ec878498b0690a3"
server_key_iv: "7fbb8be65de0788208e37f2bd9140462"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.208195 14502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:02:33.210568 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.211737 14502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:02:33.211865 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "ae364666861848eeb06c503416b6a4ca"
format_stamp: "Formatted at 2026-06-20 09:02:33 on dist-test-slave-scsf"
server_key: "f2b675d8d6acbd666ec878498b0690a3"
server_key_iv: "7fbb8be65de0788208e37f2bd9140462"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.211970 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:33.226866 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:33.229846 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:33.230062 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:33.230635 14502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:33.231586 14502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:33.231756 14502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.231830 14502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:33.231885 14502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.240556 14502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41325
I20260620 09:02:33.240590 14630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41325 every 8 connection(s)
I20260620 09:02:33.241425 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:33.242785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14502
I20260620 09:02:33.242913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:02:33.243264 392 external_mini_cluster.cc:1428] Setting key d89c5ff2fc86974c44e25263a12cba89
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:33.255065 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.243189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55827 (local address 127.0.98.62:45337)
0620 09:02:33.243346 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.243351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.244426 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 09:02:33.245938 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.245941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.245995 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.246072 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:33.247176 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.247717 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.248948 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.249186 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.249300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.251655 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:33.251659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:33.251663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:33.251689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:33.253087 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.253586 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.253592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.253594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.253677 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.254015 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.254020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.254022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.254200 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:33.254301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.254784 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.254902 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:02:33.255790 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.243641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:55827)
0620 09:02:33.244290 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.244340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.245730 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.246242 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.246249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.246584 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:33.247039 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.247051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.247845 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.247849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.248826 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.248835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.249330 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.249346 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.249540 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.250079 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:33.250094 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:33.251493 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:33.253203 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.253209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.253221 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.253481 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.253776 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.253780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.253782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.253919 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.254308 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:33.254316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:33.254559 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.254736 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.255031 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":147,"threads_started":1}
I20260620 09:02:33.256839 14631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:33.257071 14631 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:33.257553 14631 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
I20260620 09:02:33.258528 14172 ts_manager.cc:194] Registered new tserver with Master: ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325)
I20260620 09:02:33.259080 14172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55827
I20260620 09:02:33.268859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:02:33.277796 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.270648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.98.62:45337)
0620 09:02:33.270802 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.270808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.270975 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:02:33.271129 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.271133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.271179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.271262 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:33.271923 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.272439 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.273287 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.273508 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.273618 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.274866 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:33.274869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:33.274872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:33.274898 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:33.276063 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.276501 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.276505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.276507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.276589 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.276865 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.276870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.276872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.277090 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:33.277262 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.277506 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.277622 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:02:33.280397 14172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:02:33.283334 14172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:02:33.297721 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.291604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41325 (local address 127.0.0.1:58270)
0620 09:02:33.292613 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.292626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.292712 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.293830 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.293836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.293854 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:33.293933 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.293937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.294890 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.294896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.296146 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.296155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297189 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.297193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.297298 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.297340 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.297443 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.297558 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":62,"thread_start_us":84,"threads_started":1}
I20260620 09:02:33.298099 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.292559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.0.98.1:46765)
0620 09:02:33.293011 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.293027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.293050 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:02:33.293127 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.293132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.293376 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.293534 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:33.293884 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.295049 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.296720 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297346 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.297459 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.297585 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.297717 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.297965 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":156,"threads_started":1}
I20260620 09:02:33.298139 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.292403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.0.98.3:41325)
0620 09:02:33.293198 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.293257 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.293280 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:02:33.293352 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.293355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.293538 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.293678 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:33.294035 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.294763 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.296281 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297046 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.297154 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.297289 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.297908 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.297995 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":131,"threads_started":1}
I20260620 09:02:33.297916 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.292355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.0.1:47056)
0620 09:02:33.292835 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.292847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.292946 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.293550 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.293557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.293602 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:33.293754 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.293762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.295206 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.295213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.296567 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.296578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297473 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.297477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.297569 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.297594 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.297683 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.297779 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":62,"spinlock_wait_cycles":9216,"thread_start_us":133,"threads_started":1}
I20260620 09:02:33.298131 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.291565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.0.1:51890)
0620 09:02:33.292014 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.292096 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.292225 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.293558 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.293561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.293596 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:33.293734 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.293741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.295279 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.295282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.296585 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.296592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297815 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.297818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.297899 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.297921 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.297972 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.298031 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":125,"spinlock_wait_cycles":4608,"thread_start_us":74,"threads_started":1}
I20260620 09:02:33.298949 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.291740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.2:32899)
0620 09:02:33.293166 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.293189 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.293209 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:02:33.293272 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.293276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.293412 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.293886 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:33.294041 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.295142 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.296705 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.297694 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.298342 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.298472 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.298710 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.298809 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":145,"threads_started":1}
I20260620 09:02:33.300910 14565 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1f87dfda446cd8020327101e8bf7d (DEFAULT_TABLE table=test-table [id=9e7e08afcec34f53986048e452cacbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:33.301141 14431 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1f87dfda446cd8020327101e8bf7d (DEFAULT_TABLE table=test-table [id=9e7e08afcec34f53986048e452cacbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:33.301939 14565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1f87dfda446cd8020327101e8bf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:33.302107 14431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1f87dfda446cd8020327101e8bf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:33.303653 14297 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1f87dfda446cd8020327101e8bf7d (DEFAULT_TABLE table=test-table [id=9e7e08afcec34f53986048e452cacbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:33.304657 14297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1f87dfda446cd8020327101e8bf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:33.308274 14648 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Bootstrap starting.
I20260620 09:02:33.310331 14649 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Bootstrap starting.
I20260620 09:02:33.310508 14648 tablet_bootstrap.cc:654] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:33.311285 14650 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap starting.
I20260620 09:02:33.311713 14648 log.cc:824] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:33.312793 14649 tablet_bootstrap.cc:654] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:33.313408 14650 tablet_bootstrap.cc:654] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:33.313771 14649 log.cc:824] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:33.313865 14648 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: No bootstrap required, opened a new log
I20260620 09:02:33.314093 14648 ts_tablet_manager.cc:1397] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:02:33.314667 14650 log.cc:824] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:33.315644 14649 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: No bootstrap required, opened a new log
I20260620 09:02:33.315847 14649 ts_tablet_manager.cc:1397] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:02:33.316790 14650 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: No bootstrap required, opened a new log
I20260620 09:02:33.316967 14650 ts_tablet_manager.cc:1397] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:02:33.317946 14648 raft_consensus.cc:348] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.318295 14648 raft_consensus.cc:374] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:33.318359 14648 raft_consensus.cc:729] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae364666861848eeb06c503416b6a4ca, State: Initialized, Role: FOLLOWER
I20260620 09:02:33.318807 14648 consensus_queue.cc:260] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.318969 14649 raft_consensus.cc:348] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.319233 14649 raft_consensus.cc:374] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:33.319298 14649 raft_consensus.cc:729] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bff167cf4642d9b10d3c63d3c884c1, State: Initialized, Role: FOLLOWER
I20260620 09:02:33.319799 14648 ts_tablet_manager.cc:1428] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260620 09:02:33.319773 14649 consensus_queue.cc:260] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.320158 14631 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
I20260620 09:02:33.320689 14650 raft_consensus.cc:348] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.320865 14649 ts_tablet_manager.cc:1428] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:02:33.320930 14650 raft_consensus.cc:374] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:33.320988 14650 raft_consensus.cc:729] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf3d5d0521e4cbf93b3211e964052e5, State: Initialized, Role: FOLLOWER
I20260620 09:02:33.321192 14497 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
I20260620 09:02:33.321413 14650 consensus_queue.cc:260] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.322198 14363 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
I20260620 09:02:33.322328 14650 ts_tablet_manager.cc:1428] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 09:02:33.329080 14364 tablet.cc:2369] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:02:33.356145 14654 raft_consensus.cc:482] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:33.356367 14654 raft_consensus.cc:504] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.357632 14654 leader_election.cc:288] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899), 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:33.363094 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.357977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47371 (local address 127.0.98.1:46765)
0620 09:02:33.358108 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.358114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.358486 (+ 372us) server_negotiation.cc:407] Connection header received
0620 09:02:33.358545 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.358547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.358599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.358695 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:33.359120 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.360027 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.361833 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.362454 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.362575 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.362636 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.362895 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.362965 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:02:33.363094 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.358081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53365 (local address 127.0.98.2:32899)
0620 09:02:33.358458 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.358466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.358485 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:02:33.358544 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.358547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.358599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.358695 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:33.359125 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.360026 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.361833 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.362483 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.362575 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.362636 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.362895 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.362966 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 09:02:33.363123 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.357866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.98.3:53365)
0620 09:02:33.357971 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.357988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.358078 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.358717 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.358719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.358750 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:33.358996 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.359002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.360154 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.360157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.361700 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.361712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.362590 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.362594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.362726 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.362785 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.362862 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.362953 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":231,"spinlock_wait_cycles":1792}
I20260620 09:02:33.363140 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.357866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.98.3:47371)
0620 09:02:33.358370 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.358381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.358458 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.358715 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.358718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.358743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:33.358991 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.358998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.360154 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.360157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.361700 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.361712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.362590 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.362594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.362723 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.362784 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.362862 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.362953 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":358,"spinlock_wait_cycles":896,"thread_start_us":119,"threads_started":1}
I20260620 09:02:33.363849 14451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1f87dfda446cd8020327101e8bf7d" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bff167cf4642d9b10d3c63d3c884c1" is_pre_election: true
I20260620 09:02:33.363912 14317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1f87dfda446cd8020327101e8bf7d" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" is_pre_election: true
I20260620 09:02:33.364184 14317 raft_consensus.cc:2454] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 0.
I20260620 09:02:33.364184 14451 raft_consensus.cc:2454] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 0.
I20260620 09:02:33.364670 14519 leader_election.cc:302] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5, ae364666861848eeb06c503416b6a4ca; no voters:
I20260620 09:02:33.365072 14654 raft_consensus.cc:2781] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:02:33.365151 14654 raft_consensus.cc:482] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:02:33.365190 14654 raft_consensus.cc:3037] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:33.366219 14654 raft_consensus.cc:504] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.366555 14654 leader_election.cc:288] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 election: Requested vote from peers 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899), 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:33.366928 14317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1f87dfda446cd8020327101e8bf7d" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
I20260620 09:02:33.367084 14317 raft_consensus.cc:3037] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:33.368284 14317 raft_consensus.cc:2454] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 1.
I20260620 09:02:33.368575 14451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1f87dfda446cd8020327101e8bf7d" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bff167cf4642d9b10d3c63d3c884c1"
I20260620 09:02:33.368628 14519 leader_election.cc:302] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5, ae364666861848eeb06c503416b6a4ca; no voters:
I20260620 09:02:33.368722 14451 raft_consensus.cc:3037] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:33.368793 14654 raft_consensus.cc:2781] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:33.369006 14654 raft_consensus.cc:686] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 1 LEADER]: Becoming Leader. State: Replica: ae364666861848eeb06c503416b6a4ca, State: Running, Role: LEADER
I20260620 09:02:33.369352 14654 consensus_queue.cc:237] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.370028 14451 raft_consensus.cc:2454] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 1.
I20260620 09:02:33.372575 14171 catalog_manager.cc:5495] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to ae364666861848eeb06c503416b6a4ca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ae364666861848eeb06c503416b6a4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } health_report { overall_health: UNKNOWN } } }
I20260620 09:02:33.394625 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.391098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.3:41325)
0620 09:02:33.391244 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.391280 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.391296 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:02:33.391383 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.391389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.391474 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.391575 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:02:33.391820 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.392362 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.393228 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.393456 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.393560 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.393871 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:02:33.394249 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:02:33.394380 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.394438 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.394491 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:02:33.400054 14451 raft_consensus.cc:1261] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Refusing update from remote peer ae364666861848eeb06c503416b6a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:33.400053 14317 raft_consensus.cc:1261] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Refusing update from remote peer ae364666861848eeb06c503416b6a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:33.400743 14658 consensus_queue.cc:1035] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:33.400929 14654 consensus_queue.cc:1035] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:33.409451 14662 mvcc.cc:205] Tried to move back new op lower bound from 7298851444319817728 to 7298851444204240896. Current Snapshot: MvccSnapshot[applied={T|T < 7298851444319817728}]
I20260620 09:02:33.410967 14661 mvcc.cc:205] Tried to move back new op lower bound from 7298851444319817728 to 7298851444204240896. Current Snapshot: MvccSnapshot[applied={T|T < 7298851444319817728 or (T in {7298851444319817728})}]
I20260620 09:02:33.415642 14172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:33.415813 14172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:33.418136 14172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Sending DeleteTablet for 3 replicas of tablet 5af1f87dfda446cd8020327101e8bf7d
I20260620 09:02:33.418903 14431 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1f87dfda446cd8020327101e8bf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51890
I20260620 09:02:33.418920 14565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1f87dfda446cd8020327101e8bf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58270
I20260620 09:02:33.419473 14671 tablet_replica.cc:330] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: stopping tablet replica
I20260620 09:02:33.419485 14297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1f87dfda446cd8020327101e8bf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47056
I20260620 09:02:33.419840 14671 raft_consensus.cc:2229] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:02:33.420002 14672 tablet_replica.cc:330] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: stopping tablet replica
I20260620 09:02:33.420085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14234
I20260620 09:02:33.420215 14671 raft_consensus.cc:2258] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:33.422752 14671 ts_tablet_manager.cc:1905] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:33.424680 14670 tablet_replica.cc:330] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: stopping tablet replica
I20260620 09:02:33.425050 14670 raft_consensus.cc:2229] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:33.425452 14670 raft_consensus.cc:2258] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:33.426728 14671 ts_tablet_manager.cc:1918] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:33.426838 14671 log.cc:1196] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/5af1f87dfda446cd8020327101e8bf7d
I20260620 09:02:33.427023 14670 ts_tablet_manager.cc:1905] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:33.427175 14671 ts_tablet_manager.cc:1939] T 5af1f87dfda446cd8020327101e8bf7d P ae364666861848eeb06c503416b6a4ca: Deleting consensus metadata
W20260620 09:02:33.428367 14157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46765 (error 108)
I20260620 09:02:33.428579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14368
I20260620 09:02:33.430262 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.429994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.0.1:47064)
0620 09:02:33.430120 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.430190 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:02:33.430374 14156 catalog_manager.cc:4841] TS ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): tablet 5af1f87dfda446cd8020327101e8bf7d (table test-table [id=9e7e08afcec34f53986048e452cacbd8]) successfully deleted
W20260620 09:02:33.430580 14157 catalog_manager.cc:4584] TS 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5af1f87dfda446cd8020327101e8bf7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46765: connect: Connection refused (error 111)
I20260620 09:02:33.435609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14502
I20260620 09:02:33.437237 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.435788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.0.1:51894)
0620 09:02:33.437059 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.437143 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1177}
I20260620 09:02:33.441316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14140
I20260620 09:02:33.447510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--webserver_interface=127.0.98.62
--webserver_port=43053
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:33.554525 14674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:33.554803 14674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:33.554891 14674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:33.558871 14674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:33.558925 14674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:33.558945 14674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:33.558961 14674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:33.558976 14674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:33.564141 14674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:33.565171 14674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:33.566179 14674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:33.578835 14681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.578843 14680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.578887 14683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:33.579176 14674 server_base.cc:1034] running on GCE node
I20260620 09:02:33.579735 14674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:33.580837 14674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:33.582020 14674 hybrid_clock.cc:648] HybridClock initialized: now 1781946153581998 us; error 38 us; skew 500 ppm
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:33.584810 14674 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:33.585664 14674 webserver.cc:460] Webserver started at http://127.0.98.62:43053/ using document root <none> and password file <none>
I20260620 09:02:33.586232 14674 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:33.586345 14674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:33.590032 14674 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 09:02:33.592028 14690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.593230 14674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:33.593365 14674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "f7c9fb88d4ce46f0955d89d611dec978"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "9860d52649773cfffab44efcdac9a3ad"
server_key_iv: "47d01d44248d600e6daed06cacf4ce79"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.593945 14674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:33.609795 14674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:33.613122 14674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:33.613327 14674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:33.619920 14674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45337
I20260620 09:02:33.620184 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45337 every 8 connection(s)
I20260620 09:02:33.620986 14674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:33.623091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14674
I20260620 09:02:33.623534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46765
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=43437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:33.625598 14743 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:02:33.627416 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap starting.
I20260620 09:02:33.644555 14743 log.cc:824] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:33.653050 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:33.653457 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap complete.
I20260620 09:02:33.657557 14743 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:33.657953 14743 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Initialized, Role: FOLLOWER
I20260620 09:02:33.658485 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:33.658599 14743 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:33.658684 14743 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:33.658777 14743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:02:33.660658 14743 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:33.660980 14743 leader_election.cc:302] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c9fb88d4ce46f0955d89d611dec978; no voters:
I20260620 09:02:33.661324 14743 leader_election.cc:288] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:02:33.661412 14747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:02:33.661612 14747 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 LEADER]: Becoming Leader. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Running, Role: LEADER
I20260620 09:02:33.661898 14747 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:33.662530 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:33.662799 14747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c9fb88d4ce46f0955d89d611dec978. Latest consensus state: current_term: 2 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:33.662916 14747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:33.663262 14751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:33.663278 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:33.663501 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:33.668860 14751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:33.669647 14751 catalog_manager.cc:1259] Loaded cluster ID: 71a80ba93aeb48ff8ed6ac6fd8fa5e2d
I20260620 09:02:33.669741 14751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:33.670835 14751 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 09:02:33.673646 14751 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 09:02:33.678092 14751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:33.678670 14751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Loaded TSK: 0
I20260620 09:02:33.679574 14751 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:02:33.742533 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:33.742812 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:33.742884 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:33.746902 14745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:33.746979 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:33.747066 14745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:33.752187 14745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43437
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:33.753240 14745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:33.754232 14745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:33.764428 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.764430 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.764433 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:33.765146 14745 server_base.cc:1034] running on GCE node
I20260620 09:02:33.765529 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:33.766147 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:33.767325 14745 hybrid_clock.cc:648] HybridClock initialized: now 1781946153767312 us; error 31 us; skew 500 ppm
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:33.770141 14745 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:33.771061 14745 webserver.cc:460] Webserver started at http://127.0.98.1:43437/ using document root <none> and password file <none>
I20260620 09:02:33.771595 14745 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:33.771662 14745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:33.775418 14745 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:33.777580 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.778822 14745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:33.778911 14745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "fee373f99902138875f26973c5639ecf"
server_key_iv: "77a4a82fdd8b26003f737f46a42611bf"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.779444 14745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:33.793763 14745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:33.796836 14745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:33.797027 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:33.797673 14745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:33.798799 14787 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:02:33.802861 14745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:02:33.802922 14745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 09:02:33.802986 14745 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:02:33.804586 14745 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:02:33.804642 14745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:33.804888 14787 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap starting.
I20260620 09:02:33.815388 14745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46765
I20260620 09:02:33.815409 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46765 every 8 connection(s)
I20260620 09:02:33.816627 14745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:33.820067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14745
I20260620 09:02:33.820573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=33873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:33.825729 14787 log.cc:824] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:33.833006 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.818786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:42821)
0620 09:02:33.819358 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.819397 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.820290 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.821530 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.821538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.821961 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:33.822399 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.822411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.823716 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.823719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.824635 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.824644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.825365 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.825371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.825561 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.826204 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:33.826224 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:33.827839 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:33.829956 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.829961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.829974 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.830265 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.830613 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.830625 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.830627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.830765 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.831323 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:33.831329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:33.831571 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.831758 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.832084 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":127,"threads_started":1}
I20260620 09:02:33.832995 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.818747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42821 (local address 127.0.98.62:45337)
0620 09:02:33.819115 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.819131 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.819531 (+ 400us) server_negotiation.cc:407] Connection header received
0620 09:02:33.820745 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.820765 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.821112 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.821472 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:33.822584 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.823541 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.824758 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.825223 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.825404 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.828038 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:33.828043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:33.828055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:33.828082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:33.829822 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.830390 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.830396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.830400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.830515 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.830862 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.830867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.830869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.831209 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:33.831351 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.831870 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.832129 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":90,"threads_started":1}
I20260620 09:02:33.835539 14895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:33.835556 14787 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:33.835829 14895 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:33.836030 14787 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap complete.
I20260620 09:02:33.836337 14895 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
I20260620 09:02:33.837459 14787 ts_tablet_manager.cc:1397] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.013s
I20260620 09:02:33.838632 14707 ts_manager.cc:194] Registered new tserver with Master: 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:33.840248 14787 raft_consensus.cc:348] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.840773 14787 raft_consensus.cc:729] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf3d5d0521e4cbf93b3211e964052e5, State: Initialized, Role: FOLLOWER
I20260620 09:02:33.841212 14787 consensus_queue.cc:260] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:33.842136 14787 ts_tablet_manager.cc:1428] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 09:02:33.842900 14707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42821
I20260620 09:02:33.845335 14895 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:02:33.851876 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.841347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.0.98.1:46765)
0620 09:02:33.841756 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:33.841766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:33.842251 (+ 485us) server_negotiation.cc:407] Connection header received
0620 09:02:33.842430 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:33.842435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:33.842619 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:33.842734 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:33.843512 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.844396 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.845364 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.845865 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.846001 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.848276 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:33.848285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:33.848296 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:33.848320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:33.849963 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.850475 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.850480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.850482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.850566 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:33.850849 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:33.850854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:33.850856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:33.851091 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:33.851246 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:33.851571 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 09:02:33.851725 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":87,"threads_started":1}
I20260620 09:02:33.851876 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:33.841311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.0.1:47080)
0620 09:02:33.841976 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:33.842019 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:33.842166 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:33.842790 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:33.842799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:33.843247 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:33.843372 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.843380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.844554 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.844558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.845247 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:33.845256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:33.845993 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:33.846010 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:33.846131 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:33.846710 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:33.846747 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:33.848142 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:33.850097 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.850105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.850111 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.850348 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.850665 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:33.850668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:33.850670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:33.850758 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:33.851220 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:33.851227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:33.851353 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 09:02:33.851492 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 09:02:33.851696 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":107,"threads_started":1}
I20260620 09:02:33.852893 14829 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1f87dfda446cd8020327101e8bf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47080
I20260620 09:02:33.853720 14904 tablet_replica.cc:330] stopping tablet replica
I20260620 09:02:33.854007 14904 raft_consensus.cc:2229] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:33.854254 14904 raft_consensus.cc:2258] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:33.855178 14904 ts_tablet_manager.cc:1905] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:33.858592 14904 ts_tablet_manager.cc:1918] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:33.858671 14904 log.cc:1196] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/5af1f87dfda446cd8020327101e8bf7d
I20260620 09:02:33.858932 14904 ts_tablet_manager.cc:1939] T 5af1f87dfda446cd8020327101e8bf7d P 2bf3d5d0521e4cbf93b3211e964052e5: Deleting consensus metadata
I20260620 09:02:33.860582 14692 catalog_manager.cc:4841] TS 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765): tablet 5af1f87dfda446cd8020327101e8bf7d (table test-table [id=9e7e08afcec34f53986048e452cacbd8]) successfully deleted
W20260620 09:02:33.938145 14899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:33.938385 14899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:33.938424 14899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:33.942412 14899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:33.942464 14899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:33.942520 14899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:33.947379 14899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33873
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:33.948257 14899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:33.949170 14899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:02:33.958366 14899 server_base.cc:1034] running on GCE node
W20260620 09:02:33.958340 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.958319 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:33.958320 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:33.959031 14899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:33.959709 14899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:33.960875 14899 hybrid_clock.cc:648] HybridClock initialized: now 1781946153960864 us; error 26 us; skew 500 ppm
Jun 20 09:02:33 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:33.963543 14899 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:33.964416 14899 webserver.cc:460] Webserver started at http://127.0.98.2:33873/ using document root <none> and password file <none>
I20260620 09:02:33.964946 14899 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:33.965015 14899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:33.968542 14899 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 09:02:33.970618 14919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:33.971897 14899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:33.972020 14899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "52bff167cf4642d9b10d3c63d3c884c1"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "3ecab00b3ed1bab11832fb476128a5d5"
server_key_iv: "e97e0cb24d653cb105f4b6677cfb7ec5"
server_key_version: "encryptionkey@0"
I20260620 09:02:33.972600 14899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:33.990549 14899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:33.993288 14899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:33.993477 14899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:33.994042 14899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:33.995056 14926 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:02:33.999125 14899 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:02:33.999181 14899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 09:02:33.999256 14899 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:02:34.000962 14899 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:02:34.001006 14899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:34.001310 14926 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Bootstrap starting.
I20260620 09:02:34.006851 14926 log.cc:824] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:34.011294 14899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260620 09:02:34.011324 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260620 09:02:34.012084 14899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:02:34.015245 14926 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:34.015682 14926 tablet_bootstrap.cc:492] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Bootstrap complete.
I20260620 09:02:34.016021 14926 ts_tablet_manager.cc:1397] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 09:02:34.018829 14926 raft_consensus.cc:348] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.019341 14926 raft_consensus.cc:729] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bff167cf4642d9b10d3c63d3c884c1, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.019778 14926 consensus_queue.cc:260] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.020479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 14899
I20260620 09:02:34.020834 14926 ts_tablet_manager.cc:1428] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 09:02:34.020918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41325
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=38741
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:34.026589 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.013915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44175 (local address 127.0.98.62:45337)
0620 09:02:34.014062 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.014067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.014725 (+ 658us) server_negotiation.cc:407] Connection header received
0620 09:02:34.015889 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.015892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.015940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.016013 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:34.017416 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.017939 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.019533 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.019825 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.019935 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.022975 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:34.022978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:34.022980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:34.023004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:34.024612 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.025100 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.025105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.025107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.025186 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.025514 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.025519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.025521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.025693 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:34.025804 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.026288 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.026410 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:02:34.027336 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.014020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:44175)
0620 09:02:34.014542 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.014594 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.015673 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.016176 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.016185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.016662 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:34.017259 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.017276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.018082 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.018086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.019420 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.019430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.019941 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.019944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.020170 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.021266 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:34.021281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:34.022803 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:34.024723 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.024728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.024753 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.024998 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.025285 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.025288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.025290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.025423 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.025802 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:34.025809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:34.026065 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.026254 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.026556 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260620 09:02:34.028470 15035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:34.028757 15035 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:34.029338 15035 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
I20260620 09:02:34.031191 14707 ts_manager.cc:194] Registered new tserver with Master: 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
I20260620 09:02:34.032366 14707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44175
I20260620 09:02:34.034891 15035 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 09:02:34.040910 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.032010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.0.1:51904)
0620 09:02:34.032162 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.032174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.032236 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.032923 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.032926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.033111 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:34.033212 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.033219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.034199 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.034204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.034838 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.034847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.035591 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.035605 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.035691 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.036173 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:34.036183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:34.037595 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:34.039375 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.039378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.039381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.039632 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.039913 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.039916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.039919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.040008 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.040395 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:34.040398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:34.040511 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.040616 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.040748 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 09:02:34.040940 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.032140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.98.2:32899)
0620 09:02:34.032480 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.032491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.032529 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:02:34.032597 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.032602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.032786 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.032917 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:34.033343 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.034080 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.034986 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.035472 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.035928 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.037721 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:34.037729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:34.037743 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:34.037763 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:34.039269 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.039745 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.039749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.039750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.039827 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.040090 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.040092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.040093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.040299 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:34.040398 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.040684 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.040817 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
I20260620 09:02:34.041929 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1f87dfda446cd8020327101e8bf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51904
I20260620 09:02:34.042809 15041 tablet_replica.cc:330] stopping tablet replica
I20260620 09:02:34.043115 15041 raft_consensus.cc:2229] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:34.043375 15041 raft_consensus.cc:2258] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:34.044482 15041 ts_tablet_manager.cc:1905] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:34.048884 15041 ts_tablet_manager.cc:1918] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:34.048977 15041 log.cc:1196] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/5af1f87dfda446cd8020327101e8bf7d
I20260620 09:02:34.049271 15041 ts_tablet_manager.cc:1939] T 5af1f87dfda446cd8020327101e8bf7d P 52bff167cf4642d9b10d3c63d3c884c1: Deleting consensus metadata
I20260620 09:02:34.050951 14693 catalog_manager.cc:4841] TS 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): tablet 5af1f87dfda446cd8020327101e8bf7d (table test-table [id=9e7e08afcec34f53986048e452cacbd8]) successfully deleted
W20260620 09:02:34.135218 15039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:34.135452 15039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:34.135492 15039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:34.139328 15039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:34.139380 15039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:34.139436 15039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:34.144337 15039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38741
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:34.145325 15039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:34.146232 15039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:34.155398 15046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.155398 15049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.155606 15047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:34.155670 15039 server_base.cc:1034] running on GCE node
I20260620 09:02:34.156142 15039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:34.156777 15039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:34.157963 15039 hybrid_clock.cc:648] HybridClock initialized: now 1781946154157953 us; error 28 us; skew 500 ppm
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:34.160715 15039 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:34.161543 15039 webserver.cc:460] Webserver started at http://127.0.98.3:38741/ using document root <none> and password file <none>
I20260620 09:02:34.162122 15039 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:34.162227 15039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:34.166064 15039 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 09:02:34.168072 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:34.169469 15039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:34.169596 15039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "ae364666861848eeb06c503416b6a4ca"
format_stamp: "Formatted at 2026-06-20 09:02:33 on dist-test-slave-scsf"
server_key: "f2b675d8d6acbd666ec878498b0690a3"
server_key_iv: "7fbb8be65de0788208e37f2bd9140462"
server_key_version: "encryptionkey@0"
I20260620 09:02:34.170177 15039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:34.201304 15039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:34.204671 15039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:34.204878 15039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:34.205480 15039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:34.206308 15039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:34.206377 15039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:34.206446 15039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:34.206492 15039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:34.215533 15039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41325
I20260620 09:02:34.215560 15169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41325 every 8 connection(s)
I20260620 09:02:34.216532 15039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:34.217177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15039
I20260620 09:02:34.226338 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.217948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.0.98.62:45337)
0620 09:02:34.218101 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.218108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.218381 (+ 273us) server_negotiation.cc:407] Connection header received
0620 09:02:34.218564 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.218568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.218621 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.218705 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:34.219633 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.220175 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.221129 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.221461 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.221573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.222983 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:34.222987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:34.222990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:34.223015 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:34.224685 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.225083 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.225087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.225089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.225177 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.225434 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.225437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.225439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.225658 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:34.225799 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.226059 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.226181 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:34.231973 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.218431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59151 (local address 127.0.98.62:45337)
0620 09:02:34.218700 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.218703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.219593 (+ 890us) server_negotiation.cc:407] Connection header received
0620 09:02:34.221839 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.221842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.221891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.221977 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:34.223187 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.223929 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.225538 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.225843 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.225949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.228723 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:34.228726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:34.228728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:34.228749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:34.230237 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.230682 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.230684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.230685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.230755 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:34.231042 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:34.231044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:34.231045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:34.231207 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:34.231282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.231759 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.231854 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260620 09:02:34.232849 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.218964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:59151)
0620 09:02:34.219458 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.219498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.221626 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.222150 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.222157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.222504 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:34.223029 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.223049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.224065 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.224069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.225432 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.225441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.226008 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.226013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.226211 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.226768 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:34.226811 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:34.228554 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:34.230336 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.230340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.230351 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.230585 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.230829 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:34.230831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:34.230832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:34.230954 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:34.231321 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:34.231326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:34.231549 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.231703 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.231947 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260620 09:02:34.233947 15170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45337
I20260620 09:02:34.234186 15170 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:34.234669 15170 heartbeater.cc:511] Master 127.0.98.62:45337 requested a full tablet report, sending...
I20260620 09:02:34.235608 14707 ts_manager.cc:194] Registered new tserver with Master: ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325)
I20260620 09:02:34.236150 14707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59151
I20260620 09:02:34.239727 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:02:34.243578 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.240322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.98.62:45337)
0620 09:02:34.240432 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.240436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.240655 (+ 219us) server_negotiation.cc:407] Connection header received
0620 09:02:34.240777 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.240779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.240859 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.240936 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:02:34.241134 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.241618 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.242376 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.242593 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.242678 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.242979 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:02:34.243295 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:02:34.243379 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.243449 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.243492 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:02:34.244182 14707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:02:34.247295 14707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:02:34.254280 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 884f2a668e3244dba1f276e7286947b4 (DEFAULT_TABLE table=test-table [id=c69f045fa2bd47d890c7f7d77e412de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:34.254570 14829 tablet_service.cc:1467] Processing CreateTablet for tablet 884f2a668e3244dba1f276e7286947b4 (DEFAULT_TABLE table=test-table [id=c69f045fa2bd47d890c7f7d77e412de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:34.254940 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884f2a668e3244dba1f276e7286947b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:34.255097 14829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884f2a668e3244dba1f276e7286947b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:34.257517 14787 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap starting.
I20260620 09:02:34.257866 14926 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Bootstrap starting.
I20260620 09:02:34.258440 14787 tablet_bootstrap.cc:654] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:34.258733 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.253517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41325 (local address 127.0.0.1:58284)
0620 09:02:34.253697 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.253714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.253801 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.254594 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.254600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.254623 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:34.254718 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.254724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.256041 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.256044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.257260 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.257270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.258291 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.258294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.258379 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.258422 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.258515 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.258588 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 09:02:34.258852 14926 tablet_bootstrap.cc:654] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:34.259110 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.253817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.3:41325)
0620 09:02:34.254104 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.254122 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.254140 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:02:34.254235 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.254240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.254447 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.254613 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:34.254815 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.255879 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.257391 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.258175 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.258292 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.258440 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.258911 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.259006 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":115,"threads_started":1}
I20260620 09:02:34.260241 14787 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: No bootstrap required, opened a new log
I20260620 09:02:34.260335 14787 ts_tablet_manager.cc:1397] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 09:02:34.260607 14926 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: No bootstrap required, opened a new log
I20260620 09:02:34.260690 14926 ts_tablet_manager.cc:1397] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:02:34.260847 14787 raft_consensus.cc:348] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.260959 14787 raft_consensus.cc:374] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:34.261004 14787 raft_consensus.cc:729] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf3d5d0521e4cbf93b3211e964052e5, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.261137 14787 consensus_queue.cc:260] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.261122 14926 raft_consensus.cc:348] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.261240 14926 raft_consensus.cc:374] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:34.261288 14926 raft_consensus.cc:729] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bff167cf4642d9b10d3c63d3c884c1, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.261435 14926 consensus_queue.cc:260] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.261870 14787 ts_tablet_manager.cc:1428] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:02:34.262333 14926 ts_tablet_manager.cc:1428] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:34.262148 15104 tablet_service.cc:1467] Processing CreateTablet for tablet 884f2a668e3244dba1f276e7286947b4 (DEFAULT_TABLE table=test-table [id=c69f045fa2bd47d890c7f7d77e412de9]), partition=RANGE (key) PARTITION UNBOUNDED
W20260620 09:02:34.263139 15036 tablet.cc:2369] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:02:34.263832 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884f2a668e3244dba1f276e7286947b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:34.270207 15182 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Bootstrap starting.
I20260620 09:02:34.272223 15182 tablet_bootstrap.cc:654] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:34.273339 15182 log.cc:824] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:34.275144 15182 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: No bootstrap required, opened a new log
I20260620 09:02:34.275303 15182 ts_tablet_manager.cc:1397] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 09:02:34.278455 15182 raft_consensus.cc:348] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.278677 15182 raft_consensus.cc:374] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:34.278714 15182 raft_consensus.cc:729] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae364666861848eeb06c503416b6a4ca, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.279098 15182 consensus_queue.cc:260] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.279668 15170 heartbeater.cc:503] Master 127.0.98.62:45337 was elected leader, sending a full tablet report...
I20260620 09:02:34.279994 15182 ts_tablet_manager.cc:1428] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260620 09:02:34.317821 14896 tablet.cc:2369] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:02:34.335470 15184 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:34.335719 15184 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.337004 15184 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899), 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:34.340885 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.337170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.98.3:50297)
0620 09:02:34.337321 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.337338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.337430 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.337755 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.337758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.337776 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:34.337892 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.337898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.338699 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.338705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.339537 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.339546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.340327 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.340331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.340512 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.340575 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.340656 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.340733 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":24}
I20260620 09:02:34.341271 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.337294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50297 (local address 127.0.98.2:32899)
0620 09:02:34.337432 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.337439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.337457 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:02:34.337540 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.337546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.337628 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.337742 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:34.338011 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.338568 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.339672 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.340162 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.340287 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.340481 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.341062 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.341148 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:02:34.341310 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.337445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38065 (local address 127.0.98.1:46765)
0620 09:02:34.337572 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.337576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.337722 (+ 146us) server_negotiation.cc:407] Connection header received
0620 09:02:34.337866 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.337869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.337928 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.338028 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:34.338302 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.338808 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.339841 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.340505 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.340612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.340777 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.341028 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.341142 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:02:34.341763 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.337289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.98.3:38065)
0620 09:02:34.337641 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.337651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.337756 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.338040 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:34.338043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:34.338058 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:34.338186 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.338192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.338924 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.338929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.339717 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:34.339725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.340669 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.340674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:34.340794 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.340838 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:02:34.341475 (+ 637us) client_negotiation.cc:240] Negotiation successful
0620 09:02:34.341574 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260620 09:02:34.341816 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884f2a668e3244dba1f276e7286947b4" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bff167cf4642d9b10d3c63d3c884c1" is_pre_election: true
I20260620 09:02:34.342186 14988 raft_consensus.cc:2454] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 0.
I20260620 09:02:34.342370 14848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884f2a668e3244dba1f276e7286947b4" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" is_pre_election: true
I20260620 09:02:34.342700 14848 raft_consensus.cc:2454] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 0.
I20260620 09:02:34.342733 15059 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca; no voters:
I20260620 09:02:34.343056 15184 raft_consensus.cc:2781] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:02:34.343158 15184 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:02:34.343204 15184 raft_consensus.cc:3037] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:34.344277 15184 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.344686 15184 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 election: Requested vote from peers 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899), 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765)
I20260620 09:02:34.344995 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884f2a668e3244dba1f276e7286947b4" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bff167cf4642d9b10d3c63d3c884c1"
I20260620 09:02:34.345118 14988 raft_consensus.cc:3037] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:34.345120 14848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884f2a668e3244dba1f276e7286947b4" candidate_uuid: "ae364666861848eeb06c503416b6a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
I20260620 09:02:34.345268 14848 raft_consensus.cc:3037] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:34.346179 14988 raft_consensus.cc:2454] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 1.
I20260620 09:02:34.346236 14848 raft_consensus.cc:2454] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae364666861848eeb06c503416b6a4ca in term 1.
I20260620 09:02:34.346442 15059 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca; no voters:
I20260620 09:02:34.346648 15184 raft_consensus.cc:2781] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:34.346827 15184 raft_consensus.cc:686] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 1 LEADER]: Becoming Leader. State: Replica: ae364666861848eeb06c503416b6a4ca, State: Running, Role: LEADER
I20260620 09:02:34.347187 15184 consensus_queue.cc:237] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.350288 14705 catalog_manager.cc:5495] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to ae364666861848eeb06c503416b6a4ca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ae364666861848eeb06c503416b6a4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } health_report { overall_health: UNKNOWN } } }
I20260620 09:02:34.383773 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.379917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.3:41325)
0620 09:02:34.380097 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:34.380101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:34.380116 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:02:34.380216 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:34.380219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:34.380284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:34.380395 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:02:34.380659 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.381174 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.381993 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:34.382237 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:34.382468 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:34.382908 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:02:34.383343 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:02:34.383478 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:34.383514 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 09:02:34.383569 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 09:02:34.389931 14848 raft_consensus.cc:1261] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Refusing update from remote peer ae364666861848eeb06c503416b6a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:34.390019 14988 raft_consensus.cc:1261] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Refusing update from remote peer ae364666861848eeb06c503416b6a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:34.390630 15186 consensus_queue.cc:1035] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:34.390892 15184 consensus_queue.cc:1035] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:34.395866 15188 mvcc.cc:205] Tried to move back new op lower bound from 7298851448374263808 to 7298851448208871424. Current Snapshot: MvccSnapshot[applied={T|T < 7298851448374263808}]
I20260620 09:02:34.396382 15189 mvcc.cc:205] Tried to move back new op lower bound from 7298851448374263808 to 7298851448208871424. Current Snapshot: MvccSnapshot[applied={T|T < 7298851448374263808}]
I20260620 09:02:34.405277 14706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:34.405433 14706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:34.407879 14706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Sending DeleteTablet for 3 replicas of tablet 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:34.408636 14828 tablet_service.cc:1514] Processing DeleteTablet for tablet 884f2a668e3244dba1f276e7286947b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47080
I20260620 09:02:34.408635 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 884f2a668e3244dba1f276e7286947b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51904
I20260620 09:02:34.408706 15104 tablet_service.cc:1514] Processing DeleteTablet for tablet 884f2a668e3244dba1f276e7286947b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260620 09:02:34.408982 15041 tablet_replica.cc:330] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: stopping tablet replica
I20260620 09:02:34.409026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14745
I20260620 09:02:34.409039 15198 tablet_replica.cc:330] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: stopping tablet replica
I20260620 09:02:34.409119 15041 raft_consensus.cc:2229] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:34.409153 15198 raft_consensus.cc:2229] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:34.409258 15041 raft_consensus.cc:2258] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:34.409265 15198 raft_consensus.cc:2258] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:34.409890 15199 tablet_replica.cc:330] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: stopping tablet replica
I20260620 09:02:34.410259 15199 raft_consensus.cc:2229] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:02:34.410367 15041 ts_tablet_manager.cc:1905] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:34.410693 15199 raft_consensus.cc:2258] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:34.412057 15199 ts_tablet_manager.cc:1905] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:34.412901 15041 ts_tablet_manager.cc:1918] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:34.412984 15041 log.cc:1196] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/884f2a668e3244dba1f276e7286947b4
I20260620 09:02:34.413259 15041 ts_tablet_manager.cc:1939] T 884f2a668e3244dba1f276e7286947b4 P 52bff167cf4642d9b10d3c63d3c884c1: Deleting consensus metadata
I20260620 09:02:34.413918 14693 catalog_manager.cc:4841] TS 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): tablet 884f2a668e3244dba1f276e7286947b4 (table test-table [id=c69f045fa2bd47d890c7f7d77e412de9]) successfully deleted
I20260620 09:02:34.414992 15199 ts_tablet_manager.cc:1918] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:34.415109 15199 log.cc:1196] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/884f2a668e3244dba1f276e7286947b4
I20260620 09:02:34.415437 15199 ts_tablet_manager.cc:1939] T 884f2a668e3244dba1f276e7286947b4 P ae364666861848eeb06c503416b6a4ca: Deleting consensus metadata
I20260620 09:02:34.416414 14691 catalog_manager.cc:4841] TS ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): tablet 884f2a668e3244dba1f276e7286947b4 (table test-table [id=c69f045fa2bd47d890c7f7d77e412de9]) successfully deleted
W20260620 09:02:34.417032 14692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46765 (error 108)
I20260620 09:02:34.417560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14899
I20260620 09:02:34.418433 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.417746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46765 (local address 127.0.0.1:47088)
0620 09:02:34.418308 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.418367 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
W20260620 09:02:34.418571 14692 catalog_manager.cc:4584] TS 2bf3d5d0521e4cbf93b3211e964052e5 (127.0.98.1:46765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 884f2a668e3244dba1f276e7286947b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46765: connect: Connection refused (error 111)
I20260620 09:02:34.426167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15039
I20260620 09:02:34.430733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 14674
I20260620 09:02:34.437007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--webserver_interface=127.0.98.62
--webserver_port=43053
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:34.545483 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:34.545733 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:34.545773 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:34.549664 15201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:34.549716 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:34.549736 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:34.549751 15201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:34.549765 15201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:34.554888 15201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:34.555891 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:34.556938 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:34.565064 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.565047 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.565071 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:34.565064 15201 server_base.cc:1034] running on GCE node
I20260620 09:02:34.565759 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:34.566789 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:34.567981 15201 hybrid_clock.cc:648] HybridClock initialized: now 1781946154567965 us; error 28 us; skew 500 ppm
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:34.570760 15201 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:34.571578 15201 webserver.cc:460] Webserver started at http://127.0.98.62:43053/ using document root <none> and password file <none>
I20260620 09:02:34.572131 15201 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:34.572223 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:34.575824 15201 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:34.577907 15216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:34.579035 15201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 09:02:34.579170 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "f7c9fb88d4ce46f0955d89d611dec978"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "9860d52649773cfffab44efcdac9a3ad"
server_key_iv: "47d01d44248d600e6daed06cacf4ce79"
server_key_version: "encryptionkey@0"
I20260620 09:02:34.579694 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:34.607771 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:34.610678 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:34.610890 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:34.617590 15201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45337
I20260620 09:02:34.617873 15268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45337 every 8 connection(s)
I20260620 09:02:34.618652 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:34.622591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15201
I20260620 09:02:34.622938 15269 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:02:34.623106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46765
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=43437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:34.625227 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap starting.
I20260620 09:02:34.642087 15269 log.cc:824] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:34.652959 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:34.653337 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978: Bootstrap complete.
I20260620 09:02:34.657321 15269 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:34.657686 15269 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.658231 15269 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:34.658351 15269 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:34.658433 15269 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:34.658548 15269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:02:34.660328 15269 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:34.660665 15269 leader_election.cc:302] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c9fb88d4ce46f0955d89d611dec978; no voters:
I20260620 09:02:34.660998 15269 leader_election.cc:288] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:02:34.661073 15273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:02:34.661254 15273 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [term 3 LEADER]: Becoming Leader. State: Replica: f7c9fb88d4ce46f0955d89d611dec978, State: Running, Role: LEADER
I20260620 09:02:34.661545 15273 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } }
I20260620 09:02:34.662195 15269 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:34.662843 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:34.663255 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:34.663142 15275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c9fb88d4ce46f0955d89d611dec978. Latest consensus state: current_term: 3 leader_uuid: "f7c9fb88d4ce46f0955d89d611dec978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c9fb88d4ce46f0955d89d611dec978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45337 } } }
I20260620 09:02:34.663594 15275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c9fb88d4ce46f0955d89d611dec978 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:34.663893 15283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:34.668938 15283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:34.669656 15283 catalog_manager.cc:1259] Loaded cluster ID: 71a80ba93aeb48ff8ed6ac6fd8fa5e2d
I20260620 09:02:34.669727 15283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 09:02:34.670776 15283 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f737299d9ad google::LogMessage::Fail() at ??:0
@ 0x7f73729a088a google::LogMessage::SendToLog() at ??:0
@ 0x7f737299d4de google::LogMessage::Flush() at ??:0
@ 0x7f737299ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f737d2dcfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f737d2ddd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f737d2da579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f737d31e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7374bef3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7374befc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7374bf13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7374be0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73762246db start_thread at ??:0
@ 0x7f737149b61f clone at ??:0
W20260620 09:02:34.760696 15271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:34.761040 15271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:34.761109 15271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:34.767138 15271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:34.767210 15271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:34.767294 15271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:34.775226 15271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43437
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:34.776535 15271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:34.777865 15271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:34.788836 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.788986 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.788836 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:34.789328 15271 server_base.cc:1034] running on GCE node
I20260620 09:02:34.789757 15271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:34.790345 15271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:34.791494 15271 hybrid_clock.cc:648] HybridClock initialized: now 1781946154791455 us; error 57 us; skew 500 ppm
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:34.795332 15271 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:34.796416 15271 webserver.cc:460] Webserver started at http://127.0.98.1:43437/ using document root <none> and password file <none>
I20260620 09:02:34.797133 15271 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:34.797233 15271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:34.801877 15271 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 09:02:34.803983 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:34.805205 15271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:34.805354 15271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "2bf3d5d0521e4cbf93b3211e964052e5"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "fee373f99902138875f26973c5639ecf"
server_key_iv: "77a4a82fdd8b26003f737f46a42611bf"
server_key_version: "encryptionkey@0"
I20260620 09:02:34.805881 15271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:34.829346 15271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:34.832589 15271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:34.832787 15271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:34.833359 15271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:34.834548 15327 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:02:34.839663 15271 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:02:34.839757 15271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 09:02:34.839836 15271 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:02:34.841485 15271 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:02:34.841553 15271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 09:02:34.841866 15327 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap starting.
I20260620 09:02:34.852041 15271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46765
I20260620 09:02:34.852062 15434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46765 every 8 connection(s)
I20260620 09:02:34.853170 15271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:34.860137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15271
I20260620 09:02:34.860636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=33873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:34.863083 15327 log.cc:824] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:34.872614 15327 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:02:34.873021 15327 tablet_bootstrap.cc:492] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Bootstrap complete.
I20260620 09:02:34.874408 15327 ts_tablet_manager.cc:1397] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.005s
I20260620 09:02:34.877393 15327 raft_consensus.cc:348] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.877869 15327 raft_consensus.cc:729] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf3d5d0521e4cbf93b3211e964052e5, State: Initialized, Role: FOLLOWER
I20260620 09:02:34.878367 15327 consensus_queue.cc:260] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:34.879415 15327 ts_tablet_manager.cc:1428] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 09:02:34.918094 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:02:34.855112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:36751)
0620 09:02:34.855601 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.855652 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:34.856566 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:34.917135 (+ 60569us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":153,"threads_started":1}
W20260620 09:02:34.918706 15435 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45337 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45337: Client connection negotiation failed: client connection to 127.0.98.62:45337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:02:34.919420 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.919156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:60069)
0620 09:02:34.919287 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.919358 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:02:34.920064 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:34.919854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:49909)
0620 09:02:34.919962 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:34.920011 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:02:34.920266 15435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:02:34.975203 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:34.975454 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:34.975497 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:34.979436 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:34.979485 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:34.979579 15438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:34.984617 15438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33873
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:34.985555 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:34.986500 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:34.995570 15448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:34.995587 15446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:34.995627 15438 server_base.cc:1034] running on GCE node
W20260620 09:02:34.995579 15445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:34.996135 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:34.996759 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:34.997913 15438 hybrid_clock.cc:648] HybridClock initialized: now 1781946154997895 us; error 32 us; skew 500 ppm
Jun 20 09:02:34 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:35.000546 15438 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:35.001374 15438 webserver.cc:460] Webserver started at http://127.0.98.2:33873/ using document root <none> and password file <none>
I20260620 09:02:35.001961 15438 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:35.002051 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:35.005565 15438 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:35.007496 15455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.008651 15438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:35.008765 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "52bff167cf4642d9b10d3c63d3c884c1"
format_stamp: "Formatted at 2026-06-20 09:02:32 on dist-test-slave-scsf"
server_key: "3ecab00b3ed1bab11832fb476128a5d5"
server_key_iv: "e97e0cb24d653cb105f4b6677cfb7ec5"
server_key_version: "encryptionkey@0"
I20260620 09:02:35.009251 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:35.031044 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:35.033773 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:35.033975 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:35.034662 15438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:35.035440 15438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:35.035485 15438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.035548 15438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:35.035580 15438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.044567 15438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260620 09:02:35.044647 15568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260620 09:02:35.045538 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:02:35.046952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15438
I20260620 09:02:35.047346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41325
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45337
--webserver_port=38741
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:02:35.049732 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.047719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:52503)
0620 09:02:35.048251 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.048393 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
W20260620 09:02:35.050194 15569 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45337 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45337: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
I20260620 09:02:35.050832 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.050545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:41007)
0620 09:02:35.050683 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.050748 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 09:02:35.051558 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.051294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:35689)
0620 09:02:35.051425 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.051484 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:02:35.051745 15569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:02:35.155805 15572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:35.156051 15572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:35.156091 15572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:35.159941 15572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:35.159994 15572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:35.160050 15572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:35.165057 15572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38741
--tserver_master_addrs=127.0.98.62:45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:35.165926 15572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:35.166828 15572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:35.176367 15578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:35.176371 15577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:35.176399 15580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:35.176380 15572 server_base.cc:1034] running on GCE node
I20260620 09:02:35.176973 15572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:35.177490 15572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:35.178646 15572 hybrid_clock.cc:648] HybridClock initialized: now 1781946155178610 us; error 48 us; skew 500 ppm
Jun 20 09:02:35 dist-test-slave-scsf krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:35.181259 15572 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:35.182034 15572 webserver.cc:460] Webserver started at http://127.0.98.3:38741/ using document root <none> and password file <none>
I20260620 09:02:35.182566 15572 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:35.182624 15572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:35.186273 15572 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260620 09:02:35.188134 15587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.189378 15572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:35.189473 15572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "ae364666861848eeb06c503416b6a4ca"
format_stamp: "Formatted at 2026-06-20 09:02:33 on dist-test-slave-scsf"
server_key: "f2b675d8d6acbd666ec878498b0690a3"
server_key_iv: "7fbb8be65de0788208e37f2bd9140462"
server_key_version: "encryptionkey@0"
I20260620 09:02:35.189998 15572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:35.202728 15572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:35.205849 15572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:35.206048 15572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:35.206609 15572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:35.207398 15572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:35.207448 15572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.207513 15572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:35.207551 15572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:35.217001 15572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41325
I20260620 09:02:35.217036 15700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41325 every 8 connection(s)
I20260620 09:02:35.217986 15572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:35.221691 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.219986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:39907)
0620 09:02:35.220905 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.221008 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":590,"threads_started":1}
W20260620 09:02:35.222144 15701 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45337 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45337: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
I20260620 09:02:35.222672 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.222470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:42925)
0620 09:02:35.222571 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.222615 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:02:35.223256 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.223079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:51773)
0620 09:02:35.223166 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.223211 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:02:35.223469 15701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:02:35.223698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15572
I20260620 09:02:35.921291 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:35.920877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:56365)
0620 09:02:35.921112 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:35.921162 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260620 09:02:36.052636 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.052233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:47269)
0620 09:02:36.052485 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:36.052547 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260620 09:02:36.224363 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.223989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:46233)
0620 09:02:36.224217 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:36.224268 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260620 09:02:36.570034 15708 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:36.570295 15708 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:36.571372 15708 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325), 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
Jun 20 09:02:36 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 09:02:36 dist-test-slave-scsf krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 09:02:36.585387 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.571612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41325 (local address 127.0.98.1:37999)
0620 09:02:36.572015 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:36.572026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:36.572102 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:36.574060 (+ 1958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:36.574067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:36.574673 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:36.575204 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:36.575219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.576490 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.576494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:36.577394 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:36.577403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.578101 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.578105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:36.578347 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:36.578999 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:36.579015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:36.580917 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:36.582954 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:36.582959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:36.582961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:36.583203 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:36.583540 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:36.583547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:36.583684 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:36.584281 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:36.584287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:36.584603 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 09:02:36.584797 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 09:02:36.585152 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":329,"spinlock_wait_cycles":10368,"thread_start_us":50,"threads_started":1}
I20260620 09:02:36.585400 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.571612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32899 (local address 127.0.98.1:44595)
0620 09:02:36.571924 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:36.571941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:36.572050 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:36.574422 (+ 2372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:36.574427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:36.574701 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:36.575203 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:36.575218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.576494 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.576502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:36.577391 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:36.577400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.578086 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.578090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:36.578349 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:36.579055 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:36.579067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:36.580558 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:36.582890 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:36.582896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:36.582909 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:36.583175 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:36.583530 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:36.583535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:36.583677 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:36.584282 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:36.584288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:36.584598 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 09:02:36.584783 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 09:02:36.585266 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":180,"spinlock_wait_cycles":3840,"thread_start_us":66,"threads_started":1}
I20260620 09:02:36.585726 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.571763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37999 (local address 127.0.98.3:41325)
0620 09:02:36.572154 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:36.572175 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:36.572217 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:02:36.573135 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:36.573144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:36.573608 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:36.573940 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:36.575389 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.576328 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.577517 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.577960 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.578157 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:36.581079 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:36.581084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:36.581099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:36.581124 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:36.582836 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583317 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:36.583323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:36.583328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:36.583442 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583785 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:36.583788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:36.583790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:36.584160 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:36.584299 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:36.585259 (+ 960us) server_negotiation.cc:299] Negotiation successful
0620 09:02:36.585544 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":67,"threads_started":1}
I20260620 09:02:36.586375 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.571763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44595 (local address 127.0.98.2:32899)
0620 09:02:36.572155 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:36.572171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:36.572217 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:02:36.573136 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:36.573144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:36.573608 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:36.573940 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:36.575382 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.576323 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.577512 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:36.577948 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:36.578156 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:36.580809 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:36.580817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:36.580836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:36.580870 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:36.582745 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583302 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:36.583308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:36.583320 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:36.583426 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:36.583797 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:36.583800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:36.583801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:36.584160 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:36.584297 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:36.585956 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0620 09:02:36.586213 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":91,"threads_started":1}
W20260620 09:02:36.588097 15321 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
W20260620 09:02:36.588466 15323 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:36.588569 15323 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5; no voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca
I20260620 09:02:36.588824 15708 raft_consensus.cc:2726] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:02:36.922250 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:36.921979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:56125)
0620 09:02:36.922113 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:36.922168 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 09:02:37.053666 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:37.053291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:34239)
0620 09:02:37.053516 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:37.053566 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260620 09:02:37.225476 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:37.225077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:41213)
0620 09:02:37.225295 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:37.225348 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260620 09:02:37.923378 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:37.922994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:43819)
0620 09:02:37.923236 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:37.923287 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260620 09:02:38.054687 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:38.054302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:51881)
0620 09:02:38.054554 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:38.054606 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260620 09:02:38.226584 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:38.226217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:45867)
0620 09:02:38.226453 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:38.226503 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260620 09:02:38.581174 15718 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:38.581331 15718 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:38.581779 15718 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325), 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
W20260620 09:02:38.582585 15323 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
W20260620 09:02:38.582698 15321 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:38.582774 15321 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5; no voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca
I20260620 09:02:38.582899 15718 raft_consensus.cc:2726] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:02:38.924512 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:38.924040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:38181)
0620 09:02:38.924327 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:38.924376 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260620 09:02:39.055928 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:39.055512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:39899)
0620 09:02:39.055770 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:39.055832 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260620 09:02:39.227735 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:39.227373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:53139)
0620 09:02:39.227607 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:39.227655 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 09:02:39.925909 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:39.925457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:35133)
0620 09:02:39.925759 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:39.925816 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260620 09:02:40.056999 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:40.056646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:33011)
0620 09:02:40.056869 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:40.056917 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260620 09:02:40.228837 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:40.228397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:49721)
0620 09:02:40.228679 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:40.228734 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260620 09:02:40.717587 15725 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:40.717728 15725 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:40.718226 15725 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325), 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
W20260620 09:02:40.718947 15323 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
W20260620 09:02:40.719048 15321 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:40.719164 15321 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5; no voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca
I20260620 09:02:40.719310 15725 raft_consensus.cc:2726] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:02:40.926968 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:40.926596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:54391)
0620 09:02:40.926824 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:40.926876 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260620 09:02:41.058074 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:41.057693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:40391)
0620 09:02:41.057937 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:41.057988 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260620 09:02:41.229816 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:41.229486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:54499)
0620 09:02:41.229690 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:41.229739 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260620 09:02:41.927970 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:41.927601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:60429)
0620 09:02:41.927828 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:41.927876 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260620 09:02:42.059194 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:42.058784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:60573)
0620 09:02:42.059058 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:42.059110 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":67,"threads_started":1}
I20260620 09:02:42.230906 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:42.230501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:59657)
0620 09:02:42.230775 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:42.230824 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":67,"threads_started":1}
I20260620 09:02:42.929042 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:42.928676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:34439)
0620 09:02:42.928904 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:42.928953 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260620 09:02:43.060293 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:43.059903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:41223)
0620 09:02:43.060151 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:43.060201 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260620 09:02:43.232054 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:43.231642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:57985)
0620 09:02:43.231908 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:43.231971 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
I20260620 09:02:43.930290 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:43.929890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:46353)
0620 09:02:43.930120 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:43.930171 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260620 09:02:44.061514 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:44.061132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:50439)
0620 09:02:44.061372 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:44.061430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260620 09:02:44.233163 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:44.232777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:36637)
0620 09:02:44.233031 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:44.233083 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260620 09:02:44.815182 15738 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:44.815325 15738 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:44.815799 15738 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325), 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
W20260620 09:02:44.816553 15321 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
W20260620 09:02:44.816823 15323 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:44.816936 15323 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5; no voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca
I20260620 09:02:44.817121 15738 raft_consensus.cc:2726] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:02:44.931547 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:44.931034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:37367)
0620 09:02:44.931401 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:44.931455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":88,"threads_started":1}
I20260620 09:02:45.062543 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:45.062189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:51629)
0620 09:02:45.062411 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:45.062459 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260620 09:02:45.234277 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:45.233841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:38861)
0620 09:02:45.234107 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:45.234161 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260620 09:02:45.932693 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:45.932248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:59703)
0620 09:02:45.932548 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:45.932600 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
I20260620 09:02:46.063772 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:46.063408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:59933)
0620 09:02:46.063632 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:46.063681 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260620 09:02:46.235360 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:46.234971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:43257)
0620 09:02:46.235215 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:46.235266 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260620 09:02:46.933790 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:46.933397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:44241)
0620 09:02:46.933632 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:46.933685 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260620 09:02:47.065053 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:47.064677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:37701)
0620 09:02:47.064910 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:47.064963 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260620 09:02:47.236641 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:47.236162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:52973)
0620 09:02:47.236491 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:47.236554 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":74,"threads_started":1}
I20260620 09:02:47.934917 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:47.934510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:34629)
0620 09:02:47.934767 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:47.934826 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260620 09:02:48.066089 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:48.065732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:44517)
0620 09:02:48.065955 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:48.066003 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 09:02:48.237728 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:48.237308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:39327)
0620 09:02:48.237566 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:48.237637 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260620 09:02:48.936091 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:48.935658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:33381)
0620 09:02:48.935948 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:48.936000 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":127,"threads_started":1}
I20260620 09:02:49.067363 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:49.066934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:52579)
0620 09:02:49.067218 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:49.067271 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260620 09:02:49.238785 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:49.238397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.3:58321)
0620 09:02:49.238641 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:49.238695 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260620 09:02:49.937409 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:49.937038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.1:59455)
0620 09:02:49.937263 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:49.937320 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260620 09:02:49.954270 15755 raft_consensus.cc:482] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:49.954386 15755 raft_consensus.cc:504] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae364666861848eeb06c503416b6a4ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41325 } } peers { permanent_uuid: "52bff167cf4642d9b10d3c63d3c884c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32899 } } peers { permanent_uuid: "2bf3d5d0521e4cbf93b3211e964052e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46765 } }
I20260620 09:02:49.954790 15755 leader_election.cc:288] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325), 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899)
W20260620 09:02:49.955531 15323 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52bff167cf4642d9b10d3c63d3c884c1 (127.0.98.2:32899): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
W20260620 09:02:49.955651 15321 leader_election.cc:341] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae364666861848eeb06c503416b6a4ca (127.0.98.3:41325): Not found: Tablet not found: 884f2a668e3244dba1f276e7286947b4
I20260620 09:02:49.955739 15321 leader_election.cc:302] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf3d5d0521e4cbf93b3211e964052e5; no voters: 52bff167cf4642d9b10d3c63d3c884c1, ae364666861848eeb06c503416b6a4ca
I20260620 09:02:49.955915 15755 raft_consensus.cc:2726] T 884f2a668e3244dba1f276e7286947b4 P 2bf3d5d0521e4cbf93b3211e964052e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:02:50.068639 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:50.068173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45337 (local address 127.0.98.2:48393)
0620 09:02:50.068425 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:50.068550 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260620 09:02:50.224558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15271
I20260620 09:02:50.230265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15438
I20260620 09:02:50.235479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15572
2026-06-20T09:02:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19012 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:02:50 dist-test-slave-scsf krb5kdc[15759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:02:50 dist-test-slave-scsf krb5kdc[15759](info): set up 2 sockets
Jun 20 09:02:50 dist-test-slave-scsf krb5kdc[15759](info): commencing operation
krb5kdc: starting...
W20260620 09:02:51.268430 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:02:51 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:02:51.387071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:51.502987 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:51.503280 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:51.503386 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:51.507293 15772 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:02:51.507388 15772 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:02:51.507416 15772 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:02:51.507447 15772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:02:51.507488 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:51.507526 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:51.507582 15772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:02:51.507623 15772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:02:51.512666 15772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:51.513657 15772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:51.514640 15772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:51.522607 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:51.522691 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:51.522615 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:51.522855 15772 server_base.cc:1034] running on GCE node
I20260620 09:02:51.523523 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:51.524642 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:51.525825 15772 hybrid_clock.cc:648] HybridClock initialized: now 1781946171525815 us; error 24 us; skew 500 ppm
Jun 20 09:02:51 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:51.528373 15772 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:02:51.529278 15772 webserver.cc:460] Webserver started at http://127.0.98.62:35505/ using document root <none> and password file <none>
I20260620 09:02:51.529824 15772 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:51.529913 15772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:51.530131 15772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:51.532086 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "ef123959f28d4fcda201433145da3260"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "fa45ee1a6df54ef87068ca05aa52d825"
server_key_iv: "429964308ae64066854293e360084f42"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.532697 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef123959f28d4fcda201433145da3260"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "fa45ee1a6df54ef87068ca05aa52d825"
server_key_iv: "429964308ae64066854293e360084f42"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.536284 15772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:02:51.538545 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:51.539698 15772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:02:51.539837 15772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "ef123959f28d4fcda201433145da3260"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "fa45ee1a6df54ef87068ca05aa52d825"
server_key_iv: "429964308ae64066854293e360084f42"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.539949 15772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:51.553399 15772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:51.555958 15772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:51.556142 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:51.562498 15772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36953
I20260620 09:02:51.562832 15839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36953 every 8 connection(s)
I20260620 09:02:51.563392 15772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:02:51.566072 15840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:51.570955 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap starting.
I20260620 09:02:51.572484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15772
I20260620 09:02:51.572605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:02:51.572908 392 external_mini_cluster.cc:1428] Setting key d06fc43047df64d25a42e02f8078f20f
I20260620 09:02:51.573544 15840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:51.574388 15840 log.cc:824] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:51.576205 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: No bootstrap required, opened a new log
I20260620 09:02:51.579527 15840 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:02:51.579813 15840 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:51.579875 15840 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef123959f28d4fcda201433145da3260, State: Initialized, Role: FOLLOWER
I20260620 09:02:51.580256 15840 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:02:51.580401 15840 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:02:51.580503 15840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:02:51.580602 15840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 09:02:51 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:51.581715 15840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:02:51.582064 15840 leader_election.cc:302] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef123959f28d4fcda201433145da3260; no voters:
I20260620 09:02:51.582361 15840 leader_election.cc:288] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:02:51.582481 15845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:51.582700 15845 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 LEADER]: Becoming Leader. State: Replica: ef123959f28d4fcda201433145da3260, State: Running, Role: LEADER
I20260620 09:02:51.583055 15845 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:02:51.583534 15840 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:02:51.584707 15847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef123959f28d4fcda201433145da3260. Latest consensus state: current_term: 1 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:02:51.584980 15847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:51.585422 15855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:02:51.586493 15846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:02:51.586603 15846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:02:51.587280 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:51.573923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.0.98.62:36953)
0620 09:02:51.574324 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:51.574337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:51.574369 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:02:51.575169 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:51.575187 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:51.575484 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:51.575798 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:51.576662 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.577590 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.578454 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.578906 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.579072 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:51.581983 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:51.581988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:51.581999 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:51.582024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:51.584034 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:51.584619 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:51.584625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:51.584631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:51.584780 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:51.585096 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:51.585103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:51.585107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:51.585653 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:51.585842 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:51.586319 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 09:02:51.586564 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260620 09:02:51.588881 15855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:02:51.593904 15855 catalog_manager.cc:1347] Generated new cluster ID: 97ad8e0d521c4de4a6b9f606d0c8b62c
I20260620 09:02:51.593968 15855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:02:51.618963 15855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:02:51.620139 15855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:02:51.630242 15855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Generated new TSK 0
I20260620 09:02:51.630854 15855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:02:51.667354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:51.802073 15866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:51.802281 15866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:51.802320 15866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:51.806214 15866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:51.806262 15866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:51.806313 15866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:02:51.811046 15866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:51.811913 15866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:51.812817 15866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:51.822021 15871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:51.822050 15866 server_base.cc:1034] running on GCE node
W20260620 09:02:51.822041 15874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:51.822047 15872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:51.822579 15866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:51.823158 15866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:51.824371 15866 hybrid_clock.cc:648] HybridClock initialized: now 1781946171824361 us; error 30 us; skew 500 ppm
Jun 20 09:02:51 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:51.826933 15866 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:02:51.827744 15866 webserver.cc:460] Webserver started at http://127.0.98.1:33997/ using document root <none> and password file <none>
I20260620 09:02:51.828282 15866 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:51.828365 15866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:51.828598 15866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:51.830534 15866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf7fe675fe3a45a39f92b4ab97654082"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "73d6070d4542100e16e820a7d467c61b"
server_key_iv: "7849abdd454752d7e5f311ce77495f33"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.831061 15866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf7fe675fe3a45a39f92b4ab97654082"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "73d6070d4542100e16e820a7d467c61b"
server_key_iv: "7849abdd454752d7e5f311ce77495f33"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.834550 15866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:02:51.836876 15881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:51.837936 15866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:51.838056 15866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "bf7fe675fe3a45a39f92b4ab97654082"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "73d6070d4542100e16e820a7d467c61b"
server_key_iv: "7849abdd454752d7e5f311ce77495f33"
server_key_version: "encryptionkey@0"
I20260620 09:02:51.838160 15866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:51.862258 15866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:51.864990 15866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:51.865180 15866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:51.865753 15866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:51.866571 15866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:51.866641 15866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:51.866711 15866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:51.866745 15866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:51.875396 15866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39879
I20260620 09:02:51.875419 15994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39879 every 8 connection(s)
I20260620 09:02:51.876233 15866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:02:51.883386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 15866
I20260620 09:02:51.883515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:02:51.883841 392 external_mini_cluster.cc:1428] Setting key 59fc2d276f683a243cc20a8dfe4dec31
Jun 20 09:02:51 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:51.889544 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:51.877988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53961 (local address 127.0.98.62:36953)
0620 09:02:51.878121 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:51.878125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:51.879104 (+ 979us) server_negotiation.cc:407] Connection header received
0620 09:02:51.879956 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:51.879959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:51.880004 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:51.880076 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:51.881294 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.881798 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.882835 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.883048 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.883124 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:51.886025 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:51.886029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:51.886031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:51.886063 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:51.887542 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:51.888072 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:51.888075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:51.888076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:51.888149 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:51.888506 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:51.888509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:51.888513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:51.888679 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:51.888789 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:51.889248 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 09:02:51.889381 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:02:51.890272 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:51.878381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:53961)
0620 09:02:51.878987 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:51.879025 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:51.879774 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:51.880240 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:51.880246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:51.880669 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:51.881148 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:51.881161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.881897 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.881899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:51.882732 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:51.882740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:51.883200 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:51.883205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:51.883444 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:51.884070 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:51.884084 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:51.885874 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:51.887659 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:51.887663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:51.887673 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:51.887949 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:51.888237 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:51.888242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:51.888244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:51.888381 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:51.888789 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:51.888797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:51.889023 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 09:02:51.889193 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 09:02:51.889491 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":113,"threads_started":1}
I20260620 09:02:51.891455 15995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:02:51.891703 15995 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:51.892237 15995 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:02:51.893730 15804 ts_manager.cc:194] Registered new tserver with Master: bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879)
I20260620 09:02:51.894908 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:02:51.909672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:52.013147 16000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:52.013366 16000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:52.013406 16000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:52.017426 16000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:52.017472 16000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:52.017527 16000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:02:52.022344 16000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:52.023178 16000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:52.024035 16000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:52.032714 16005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:52.032711 16006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:52.032711 16008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:52.032788 16000 server_base.cc:1034] running on GCE node
I20260620 09:02:52.033303 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:52.033850 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:52.035007 16000 hybrid_clock.cc:648] HybridClock initialized: now 1781946172034998 us; error 38 us; skew 500 ppm
Jun 20 09:02:52 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:52.037710 16000 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:02:52.038568 16000 webserver.cc:460] Webserver started at http://127.0.98.2:45177/ using document root <none> and password file <none>
I20260620 09:02:52.039076 16000 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:52.039163 16000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:52.039386 16000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:52.041379 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "c17895d23e774900bb261801ee5c0dc3"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "3b1a394fce97fde32940b9a04b78bd74"
server_key_iv: "1ad65d5e762cdec00e4daa5e6f6846dc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.041918 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c17895d23e774900bb261801ee5c0dc3"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "3b1a394fce97fde32940b9a04b78bd74"
server_key_iv: "1ad65d5e762cdec00e4daa5e6f6846dc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.045504 16000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:02:52.047654 16015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.048810 16000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:02:52.048933 16000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "c17895d23e774900bb261801ee5c0dc3"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "3b1a394fce97fde32940b9a04b78bd74"
server_key_iv: "1ad65d5e762cdec00e4daa5e6f6846dc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.049033 16000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:52.071197 16000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:52.074182 16000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:52.074378 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:52.074998 16000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:52.075742 16000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:52.075809 16000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.075873 16000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:52.075914 16000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.084071 16000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35531
I20260620 09:02:52.084126 16128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35531 every 8 connection(s)
I20260620 09:02:52.085089 16000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:02:52 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:52.095196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16000
I20260620 09:02:52.095343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:02:52.095588 392 external_mini_cluster.cc:1428] Setting key 11301365e4bdd7c9036a938a6152975e
I20260620 09:02:52.098162 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.086859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35677 (local address 127.0.98.62:36953)
0620 09:02:52.086974 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.086978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.088112 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 09:02:52.089013 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.089016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.089071 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.089141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:52.090206 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.090747 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.091852 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.092083 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.092195 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.094531 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:52.094534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:52.094537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:52.094565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:52.096059 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.096615 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.096619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.096621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.096702 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.097083 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.097085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.097087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.097248 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:52.097330 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.097928 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.098024 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:02:52.099193 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.087186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:35677)
0620 09:02:52.087983 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.088020 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.088817 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.089315 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.089322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.089636 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:52.090059 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.090071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.090891 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.090897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.091729 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.091738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.092229 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.092233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.092398 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.093006 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:52.093021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:52.094367 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:52.096172 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:52.096178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:52.096190 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:52.096524 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:52.096800 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:52.096806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:52.096808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:52.096994 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:52.097369 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:52.097377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:52.097676 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.097877 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.098254 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":500,"threads_started":1}
I20260620 09:02:52.100401 16129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:02:52.100663 16129 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:52.101126 16129 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
I20260620 09:02:52.102085 15804 ts_manager.cc:194] Registered new tserver with Master: c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
I20260620 09:02:52.102630 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35677
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:02:52.121271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:02:52.229075 16134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:02:52.229302 16134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:02:52.229342 16134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:02:52.233280 16134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:02:52.233327 16134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:02:52.233426 16134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:02:52.239038 16134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:02:52.239979 16134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:02:52.241009 16134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:02:52.250180 16139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:02:52.250159 16140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:52.250249 16134 server_base.cc:1034] running on GCE node
W20260620 09:02:52.250156 16142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:02:52.250846 16134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:02:52.251434 16134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:02:52.252652 16134 hybrid_clock.cc:648] HybridClock initialized: now 1781946172252626 us; error 47 us; skew 500 ppm
Jun 20 09:02:52 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:02:52.255334 16134 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:02:52.256134 16134 webserver.cc:460] Webserver started at http://127.0.98.3:42761/ using document root <none> and password file <none>
I20260620 09:02:52.256773 16134 fs_manager.cc:359] Metadata directory not provided
I20260620 09:02:52.256862 16134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:02:52.257087 16134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:02:52.259006 16134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "43257bc3874a4a9d883cf10bdb610f91"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "0d051bedd94de73c96c797e6d11fafff"
server_key_iv: "7c87da4f4a16f9c903dab455123debdc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.259558 16134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43257bc3874a4a9d883cf10bdb610f91"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "0d051bedd94de73c96c797e6d11fafff"
server_key_iv: "7c87da4f4a16f9c903dab455123debdc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.263041 16134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 09:02:52.265426 16149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.266537 16134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:02:52.266666 16134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "43257bc3874a4a9d883cf10bdb610f91"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "0d051bedd94de73c96c797e6d11fafff"
server_key_iv: "7c87da4f4a16f9c903dab455123debdc"
server_key_version: "encryptionkey@0"
I20260620 09:02:52.266772 16134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:02:52.297307 16134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:02:52.300657 16134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:02:52.300855 16134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:02:52.301467 16134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:02:52.302289 16134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:02:52.302358 16134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.302435 16134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:02:52.302476 16134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:02:52.310997 16134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34803
I20260620 09:02:52.311028 16262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34803 every 8 connection(s)
I20260620 09:02:52.311828 16134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:02:52.316967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16134
I20260620 09:02:52.317107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:02:52.317389 392 external_mini_cluster.cc:1428] Setting key 272f31c7f367cd16bcedbdccfb3585d5
Jun 20 09:02:52 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:02:52.325953 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.313587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33215 (local address 127.0.98.62:36953)
0620 09:02:52.313723 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.313726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.314505 (+ 779us) server_negotiation.cc:407] Connection header received
0620 09:02:52.315737 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.315740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.315799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.315880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:52.317206 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.317878 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.319361 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.319613 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.319695 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.322687 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:52.322690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:52.322692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:52.322716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:52.324184 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.324687 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.324690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.324691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.324778 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.325050 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.325052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.325053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.325201 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:52.325273 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.325751 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.325845 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:02:52.326685 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.313943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:33215)
0620 09:02:52.314361 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.314404 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.315531 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.316045 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.316051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.316386 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:02:52.316990 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.317012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.318002 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.318007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.319221 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.319233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.319768 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.319772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.320035 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.321057 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:02:52.321077 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:02:52.322524 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:02:52.324276 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:52.324280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:52.324290 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:52.324610 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:52.324853 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:02:52.324855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:02:52.324856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:02:52.324975 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:02:52.325280 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:02:52.325284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:02:52.325490 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.325640 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.325855 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260620 09:02:52.327796 16263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:02:52.327993 16263 heartbeater.cc:464] Registering TS with master...
I20260620 09:02:52.328388 16263 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
I20260620 09:02:52.329353 15804 ts_manager.cc:194] Registered new tserver with Master: 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803)
I20260620 09:02:52.329948 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33215
I20260620 09:02:52.331079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:02:52.339437 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.332968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.0.98.62:36953)
0620 09:02:52.333070 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.333073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.333246 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:02:52.333324 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.333326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.333359 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.333422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:02:52.334031 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.334493 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.335238 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.335434 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.335506 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.336794 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:02:52.336797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:02:52.336799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:02:52.336819 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:02:52.338174 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.338530 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.338533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.338534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.338601 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:02:52.338820 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:02:52.338822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:02:52.338823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:02:52.339001 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:02:52.339103 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.339257 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.339338 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:02:52.342146 15804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:02:52.345381 15804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:02:52.359143 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.354045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.98.1:39879)
0620 09:02:52.354716 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.354734 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.354756 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:02:52.354828 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.354833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.355003 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.355163 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:52.355388 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.356549 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.357963 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.358530 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.358634 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.358776 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.358917 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.358999 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":76,"threads_started":1}
I20260620 09:02:52.359153 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.353862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39879 (local address 127.0.0.1:42672)
0620 09:02:52.354369 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.354382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.354471 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.355159 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.355163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.355181 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:52.355267 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.355273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.356676 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.356679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.357856 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.357862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.358645 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.358649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.358744 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.358778 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.358893 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.359004 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":18,"thread_start_us":91,"threads_started":1}
I20260620 09:02:52.359362 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.353181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.0.1:48300)
0620 09:02:52.354067 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.354112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.354278 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.354892 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.354898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.354921 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:52.355063 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.355074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.356234 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.356238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.357637 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.357647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.358883 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.358888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.358975 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.358994 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.359058 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.359111 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":47,"thread_start_us":72,"threads_started":1}
I20260620 09:02:52.359469 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.353333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.0.98.2:35531)
0620 09:02:52.353638 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.353648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.354428 (+ 780us) server_negotiation.cc:407] Connection header received
0620 09:02:52.354518 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.354522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.354734 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.354907 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:52.355174 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.356123 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.357758 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.358581 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.358674 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.358806 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.359231 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.359326 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":122,"threads_started":1}
I20260620 09:02:52.361135 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.353322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34803 (local address 127.0.0.1:41312)
0620 09:02:52.354295 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.354310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.354383 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.355874 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.355877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.355895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:52.355967 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.355972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.357250 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.357253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.358431 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.358443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.360005 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.360009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.360843 (+ 834us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.360867 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.360948 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.361012 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260620 09:02:52.361521 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.353929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.0.98.3:34803)
0620 09:02:52.355465 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.355477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.355495 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:02:52.355559 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.355564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.355725 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.355871 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:52.356077 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.357111 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.358563 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.359884 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.359988 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.360103 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.361002 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.361088 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436,"thread_start_us":84,"threads_started":1}
I20260620 09:02:52.362155 15929 tablet_service.cc:1467] Processing CreateTablet for tablet 1846e32cb790484e88e0357abc0a7b3d (DEFAULT_TABLE table=test-table [id=43f199d9f75e49a3bbffc9d55db72024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:52.362141 16063 tablet_service.cc:1467] Processing CreateTablet for tablet 1846e32cb790484e88e0357abc0a7b3d (DEFAULT_TABLE table=test-table [id=43f199d9f75e49a3bbffc9d55db72024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:52.363003 15929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1846e32cb790484e88e0357abc0a7b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:52.363104 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1846e32cb790484e88e0357abc0a7b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:52.363821 16197 tablet_service.cc:1467] Processing CreateTablet for tablet 1846e32cb790484e88e0357abc0a7b3d (DEFAULT_TABLE table=test-table [id=43f199d9f75e49a3bbffc9d55db72024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:02:52.364748 16197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1846e32cb790484e88e0357abc0a7b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:02:52.369540 16280 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap starting.
I20260620 09:02:52.370424 16281 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Bootstrap starting.
I20260620 09:02:52.372524 16282 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Bootstrap starting.
I20260620 09:02:52.372540 16280 tablet_bootstrap.cc:654] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:52.373068 16281 tablet_bootstrap.cc:654] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:52.373862 16280 log.cc:824] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:52.373862 16281 log.cc:824] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:52.374359 16282 tablet_bootstrap.cc:654] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Neither blocks nor log segments found. Creating new log.
I20260620 09:02:52.375581 16282 log.cc:824] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Log is configured to *not* fsync() on all Append() calls
I20260620 09:02:52.375816 16280 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: No bootstrap required, opened a new log
I20260620 09:02:52.375816 16281 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: No bootstrap required, opened a new log
I20260620 09:02:52.376003 16280 ts_tablet_manager.cc:1397] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 09:02:52.376013 16281 ts_tablet_manager.cc:1397] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:02:52.377470 16282 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: No bootstrap required, opened a new log
I20260620 09:02:52.377688 16282 ts_tablet_manager.cc:1397] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:02:52.379892 16280 raft_consensus.cc:348] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.379892 16281 raft_consensus.cc:348] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.380153 16281 raft_consensus.cc:374] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:52.380153 16280 raft_consensus.cc:374] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:52.380203 16280 raft_consensus.cc:729] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fe675fe3a45a39f92b4ab97654082, State: Initialized, Role: FOLLOWER
I20260620 09:02:52.380203 16281 raft_consensus.cc:729] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17895d23e774900bb261801ee5c0dc3, State: Initialized, Role: FOLLOWER
I20260620 09:02:52.380720 16281 consensus_queue.cc:260] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.380725 16280 consensus_queue.cc:260] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.380750 16282 raft_consensus.cc:348] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.381035 16282 raft_consensus.cc:374] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:02:52.381098 16282 raft_consensus.cc:729] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43257bc3874a4a9d883cf10bdb610f91, State: Initialized, Role: FOLLOWER
I20260620 09:02:52.381572 16282 consensus_queue.cc:260] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.381696 16281 ts_tablet_manager.cc:1428] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 09:02:52.381696 16280 ts_tablet_manager.cc:1428] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 09:02:52.382040 15995 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
I20260620 09:02:52.382334 16263 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
I20260620 09:02:52.382599 16282 ts_tablet_manager.cc:1428] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 09:02:52.383183 16129 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
I20260620 09:02:52.389730 16287 raft_consensus.cc:482] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:02:52.389870 16287 raft_consensus.cc:504] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.390805 16287 leader_election.cc:288] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531), 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803)
I20260620 09:02:52.394701 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.391059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39669 (local address 127.0.98.2:35531)
0620 09:02:52.391164 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.391168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.391181 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:02:52.391297 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.391300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.391344 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.391421 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:52.391703 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.392367 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.393665 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.394140 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.394224 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.394272 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.394510 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.394575 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:02:52.394726 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.391017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.98.1:39669)
0620 09:02:52.391112 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.391126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.391207 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.391446 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.391449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.391466 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:52.391563 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.391568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.392520 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.392523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.393535 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.393545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.394253 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.394257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.394370 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.394404 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.394486 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.394557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 09:02:52.394953 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.391200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35805 (local address 127.0.98.3:34803)
0620 09:02:52.391373 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.391380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.391641 (+ 261us) server_negotiation.cc:407] Connection header received
0620 09:02:52.391762 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.391764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.391803 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.391864 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:02:52.392121 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.392819 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.393865 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.394376 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.394497 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.394554 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.394743 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.394820 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 09:02:52.394966 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.391025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34803 (local address 127.0.98.1:35805)
0620 09:02:52.391538 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.391550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:02:52.391635 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:02:52.391892 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:02:52.391895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:02:52.391913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:02:52.392019 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.392025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.392922 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.392924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.393744 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:02:52.393755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.394513 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.394518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:02:52.394612 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.394634 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:02:52.394720 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:02:52.394791 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":25,"thread_start_us":102,"threads_started":1}
I20260620 09:02:52.395366 16083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1846e32cb790484e88e0357abc0a7b3d" candidate_uuid: "bf7fe675fe3a45a39f92b4ab97654082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17895d23e774900bb261801ee5c0dc3" is_pre_election: true
I20260620 09:02:52.395633 16083 raft_consensus.cc:2454] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7fe675fe3a45a39f92b4ab97654082 in term 0.
I20260620 09:02:52.395620 16217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1846e32cb790484e88e0357abc0a7b3d" candidate_uuid: "bf7fe675fe3a45a39f92b4ab97654082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43257bc3874a4a9d883cf10bdb610f91" is_pre_election: true
I20260620 09:02:52.395893 16217 raft_consensus.cc:2454] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7fe675fe3a45a39f92b4ab97654082 in term 0.
I20260620 09:02:52.396080 15884 leader_election.cc:302] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082, c17895d23e774900bb261801ee5c0dc3; no voters:
I20260620 09:02:52.396313 16287 raft_consensus.cc:2781] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:02:52.396390 16287 raft_consensus.cc:482] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:02:52.396432 16287 raft_consensus.cc:3037] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:52.397357 16287 raft_consensus.cc:504] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.397666 16287 leader_election.cc:288] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 1 election: Requested vote from peers c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531), 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803)
I20260620 09:02:52.398001 16083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1846e32cb790484e88e0357abc0a7b3d" candidate_uuid: "bf7fe675fe3a45a39f92b4ab97654082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17895d23e774900bb261801ee5c0dc3"
I20260620 09:02:52.398038 16217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1846e32cb790484e88e0357abc0a7b3d" candidate_uuid: "bf7fe675fe3a45a39f92b4ab97654082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43257bc3874a4a9d883cf10bdb610f91"
I20260620 09:02:52.398104 16083 raft_consensus.cc:3037] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:52.398185 16217 raft_consensus.cc:3037] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:02:52.399091 16083 raft_consensus.cc:2454] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7fe675fe3a45a39f92b4ab97654082 in term 1.
I20260620 09:02:52.399242 16217 raft_consensus.cc:2454] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7fe675fe3a45a39f92b4ab97654082 in term 1.
I20260620 09:02:52.399367 15884 leader_election.cc:302] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082, c17895d23e774900bb261801ee5c0dc3; no voters:
I20260620 09:02:52.399534 16287 raft_consensus.cc:2781] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:02:52.399747 16287 raft_consensus.cc:686] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 LEADER]: Becoming Leader. State: Replica: bf7fe675fe3a45a39f92b4ab97654082, State: Running, Role: LEADER
I20260620 09:02:52.400039 16287 consensus_queue.cc:237] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:02:52.402848 15803 catalog_manager.cc:5495] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bf7fe675fe3a45a39f92b4ab97654082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } health_report { overall_health: HEALTHY } } }
I20260620 09:02:52.421289 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.417007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.98.1:39879)
0620 09:02:52.417119 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:02:52.417124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:02:52.417138 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:02:52.417319 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:02:52.417322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:02:52.417382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:02:52.417487 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:02:52.417866 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.418490 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.419771 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:02:52.420143 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:02:52.420249 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:02:52.420595 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:02:52.420982 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:02:52.421073 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:02:52.421130 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:02:52.421174 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:02:52.426651 16217 raft_consensus.cc:1261] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Refusing update from remote peer bf7fe675fe3a45a39f92b4ab97654082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:52.427248 16083 raft_consensus.cc:1261] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Refusing update from remote peer bf7fe675fe3a45a39f92b4ab97654082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:02:52.427323 16290 consensus_queue.cc:1035] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:52.427831 16290 consensus_queue.cc:1035] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:02:52.444343 15803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:52.444550 15803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:02:52.446933 15803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Sending DeleteTablet for 3 replicas of tablet 1846e32cb790484e88e0357abc0a7b3d
I20260620 09:02:52.447628 16197 tablet_service.cc:1514] Processing DeleteTablet for tablet 1846e32cb790484e88e0357abc0a7b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41312
I20260620 09:02:52.447754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15866
I20260620 09:02:52.447746 16063 tablet_service.cc:1514] Processing DeleteTablet for tablet 1846e32cb790484e88e0357abc0a7b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48300
I20260620 09:02:52.451753 16303 tablet_replica.cc:330] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: stopping tablet replica
I20260620 09:02:52.452023 16304 tablet_replica.cc:330] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: stopping tablet replica
I20260620 09:02:52.452037 16303 raft_consensus.cc:2229] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:52.452297 16304 raft_consensus.cc:2229] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:02:52.452344 16303 raft_consensus.cc:2258] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:02:52.452582 16304 raft_consensus.cc:2258] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:02:52.453194 15790 connection.cc:535] client connection to 127.0.98.1:39879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:02:52.453233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16000
W20260620 09:02:52.453338 15790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:02:52.454608 16303 ts_tablet_manager.cc:1905] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:02:52.455415 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.455118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39879 (local address 127.0.0.1:42680)
0620 09:02:52.455261 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.455328 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 09:02:52.455580 15790 catalog_manager.cc:4584] TS bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1846e32cb790484e88e0357abc0a7b3d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39879: connect: Connection refused (error 111)
I20260620 09:02:52.457059 16303 ts_tablet_manager.cc:1918] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:02:52.457168 16303 log.cc:1196] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/1846e32cb790484e88e0357abc0a7b3d
I20260620 09:02:52.457549 16303 ts_tablet_manager.cc:1939] T 1846e32cb790484e88e0357abc0a7b3d P 43257bc3874a4a9d883cf10bdb610f91: Deleting consensus metadata
I20260620 09:02:52.458570 15791 catalog_manager.cc:4841] TS 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): tablet 1846e32cb790484e88e0357abc0a7b3d (table test-table [id=43f199d9f75e49a3bbffc9d55db72024]) successfully deleted
I20260620 09:02:52.459619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16134
I20260620 09:02:52.460376 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:02:52.459741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.0.1:48312)
0620 09:02:52.460213 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 09:02:52.460286 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
I20260620 09:02:52.465226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 15772
I20260620 09:03:12.471551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--webserver_interface=127.0.98.62
--webserver_port=35505
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:12.580621 16306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:12.580861 16306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:12.580925 16306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:12.585464 16306 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:03:12.585510 16306 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:03:12.585526 16306 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:03:12.585539 16306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:12.585577 16306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:12.585610 16306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:12.585644 16306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:12.585675 16306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:12.590917 16306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:12.592051 16306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:12.593441 16306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:12.607370 16312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:12.607370 16313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:12.607424 16306 server_base.cc:1034] running on GCE node
W20260620 09:03:12.607421 16315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:12.608227 16306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:12.609404 16306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:12.610596 16306 hybrid_clock.cc:648] HybridClock initialized: now 1781946192610584 us; error 32 us; skew 500 ppm
Jun 20 09:03:12 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:12.613401 16306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:03:12.614269 16306 webserver.cc:460] Webserver started at http://127.0.98.62:35505/ using document root <none> and password file <none>
I20260620 09:03:12.614845 16306 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:12.614967 16306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:12.618822 16306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:12.620931 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:12.622110 16306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:12.622258 16306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "ef123959f28d4fcda201433145da3260"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "fa45ee1a6df54ef87068ca05aa52d825"
server_key_iv: "429964308ae64066854293e360084f42"
server_key_version: "encryptionkey@0"
I20260620 09:03:12.622862 16306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:12.643211 16306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:12.646461 16306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:12.646659 16306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:12.653487 16306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36953
I20260620 09:03:12.653761 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36953 every 8 connection(s)
I20260620 09:03:12.654596 16306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:12.657820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16306
I20260620 09:03:12.658257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39879
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=33997
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:03:12.659152 16375 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:03:12.660833 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap starting.
I20260620 09:03:12.677342 16375 log.cc:824] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:12.685598 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:03:12.685992 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap complete.
I20260620 09:03:12.690006 16375 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:12.690382 16375 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef123959f28d4fcda201433145da3260, State: Initialized, Role: FOLLOWER
I20260620 09:03:12.690872 16375 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:12.691015 16375 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:12.691102 16375 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:12.691212 16375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:03:12.693691 16375 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:12.694077 16375 leader_election.cc:302] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef123959f28d4fcda201433145da3260; no voters:
I20260620 09:03:12.694432 16375 leader_election.cc:288] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:03:12.694646 16379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:03:12.694984 16379 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 LEADER]: Becoming Leader. State: Replica: ef123959f28d4fcda201433145da3260, State: Running, Role: LEADER
I20260620 09:03:12.695431 16379 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:12.695685 16375 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:12.696874 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef123959f28d4fcda201433145da3260. Latest consensus state: current_term: 2 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:03:12.697091 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:12.697209 16380 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:03:12.697327 16380 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:12.697606 16388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:12.702657 16388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:12.703542 16388 catalog_manager.cc:1259] Loaded cluster ID: 97ad8e0d521c4de4a6b9f606d0c8b62c
I20260620 09:03:12.703604 16388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:12.705936 16388 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:12.706607 16388 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Loaded TSK: 0
I20260620 09:03:12.731909 16388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Generated new TSK 1
I20260620 09:03:12.732213 16388 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:03:12.776487 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:12.776731 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:12.776772 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:12.780799 16377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:12.780850 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:12.780941 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:12.785992 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33997
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:12.787063 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:12.788053 16377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:12.798256 16403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:12.798321 16402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:12.798323 16405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:12.798899 16377 server_base.cc:1034] running on GCE node
I20260620 09:03:12.799309 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:12.799909 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:12.801112 16377 hybrid_clock.cc:648] HybridClock initialized: now 1781946192801080 us; error 43 us; skew 500 ppm
Jun 20 09:03:12 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:12.804016 16377 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:03:12.805053 16377 webserver.cc:460] Webserver started at http://127.0.98.1:33997/ using document root <none> and password file <none>
I20260620 09:03:12.805735 16377 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:12.805850 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:12.810761 16377 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:12.813326 16412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:12.814823 16377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:03:12.814944 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "bf7fe675fe3a45a39f92b4ab97654082"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "73d6070d4542100e16e820a7d467c61b"
server_key_iv: "7849abdd454752d7e5f311ce77495f33"
server_key_version: "encryptionkey@0"
I20260620 09:03:12.815630 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:12.840953 16377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:12.844805 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:12.845005 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:12.845707 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:12.846910 16419 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:03:12.850900 16377 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:03:12.850968 16377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 09:03:12.851027 16377 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:03:12.853014 16377 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:03:12.853067 16377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:12.853375 16419 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap starting.
I20260620 09:03:12.864135 16377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39879
I20260620 09:03:12.864229 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39879 every 8 connection(s)
I20260620 09:03:12.865152 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:12.871942 16419 log.cc:824] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:12.874792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16377
I20260620 09:03:12.875365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35531
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=45177
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:03:12 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:12.879869 16419 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:03:12.880232 16419 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap complete.
I20260620 09:03:12.881523 16419 ts_tablet_manager.cc:1397] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.014s
I20260620 09:03:12.882647 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:12.867082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:50865)
0620 09:03:12.867643 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:12.867699 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:12.868962 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:12.870069 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:12.870078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:12.870552 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:12.871174 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:12.871194 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.872611 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.872614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:12.873853 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:12.873863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.874582 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.874586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:12.874766 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:12.875464 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:12.875485 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:12.877293 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:12.879390 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:12.879404 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:12.879416 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:12.879670 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:12.880051 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:12.880057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:12.880062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:12.880228 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:12.880932 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:12.880940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:12.881255 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 09:03:12.881451 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 09:03:12.881765 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":142,"threads_started":1}
I20260620 09:03:12.884184 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:12.867222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50865 (local address 127.0.98.62:36953)
0620 09:03:12.867656 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:12.867676 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:12.867829 (+ 153us) server_negotiation.cc:407] Connection header received
0620 09:03:12.869362 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:12.869373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:12.869665 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:12.869945 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:12.871346 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.872448 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.873982 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.874428 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.874602 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:12.877517 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:12.877521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:12.877532 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:12.877558 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:12.879257 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:12.879785 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:12.879797 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:12.879801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:12.879935 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:12.880336 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:12.880341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:12.880346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:12.880803 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:12.880947 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:12.883083 (+ 2136us) server_negotiation.cc:299] Negotiation successful
0620 09:03:12.883381 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":111,"threads_started":1}
I20260620 09:03:12.884318 16419 raft_consensus.cc:348] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:03:12.884828 16419 raft_consensus.cc:729] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fe675fe3a45a39f92b4ab97654082, State: Initialized, Role: FOLLOWER
I20260620 09:03:12.885308 16419 consensus_queue.cc:260] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:03:12.886291 16419 ts_tablet_manager.cc:1428] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:03:12.886579 16527 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:03:12.886823 16527 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:12.887401 16527 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
I20260620 09:03:12.889953 16339 ts_manager.cc:194] Registered new tserver with Master: bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879)
I20260620 09:03:12.893253 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50865
I20260620 09:03:12.895843 16527 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
Jun 20 09:03:12 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:03:12.903682 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:12.892578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39879 (local address 127.0.0.1:53864)
0620 09:03:12.893093 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:12.893136 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:12.893257 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:12.893817 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:12.893824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:12.894239 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:12.894365 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:12.894375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.895751 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.895755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:12.896845 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:12.896857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.897779 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.897794 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:12.897908 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:12.898501 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:12.898518 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:12.899992 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:12.901939 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:12.901949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:12.901954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:12.902183 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:12.902490 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:12.902493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:12.902495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:12.902583 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:12.903031 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:12.903038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:12.903160 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 09:03:12.903297 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 09:03:12.903496 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":108,"threads_started":1}
I20260620 09:03:12.903684 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:12.892631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.98.1:39879)
0620 09:03:12.893269 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:12.893282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:12.893305 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:03:12.893468 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:12.893473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:12.893659 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:12.893795 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:12.894498 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.895641 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.896974 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:12.897677 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:12.897770 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:12.900125 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:12.900134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:12.900145 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:12.900168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:12.901804 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:12.902297 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:12.902302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:12.902304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:12.902386 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:12.902681 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:12.902684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:12.902686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:12.902905 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:12.903055 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:12.903388 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 09:03:12.903540 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":549,"thread_start_us":99,"threads_started":1}
I20260620 09:03:12.904743 16461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1846e32cb790484e88e0357abc0a7b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53864
I20260620 09:03:12.905550 16536 tablet_replica.cc:330] stopping tablet replica
I20260620 09:03:12.905848 16536 raft_consensus.cc:2229] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:12.906096 16536 raft_consensus.cc:2258] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:12.907167 16536 ts_tablet_manager.cc:1905] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:12.911285 16536 ts_tablet_manager.cc:1918] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:12.911377 16536 log.cc:1196] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal/wals/1846e32cb790484e88e0357abc0a7b3d
I20260620 09:03:12.911639 16536 ts_tablet_manager.cc:1939] T 1846e32cb790484e88e0357abc0a7b3d P bf7fe675fe3a45a39f92b4ab97654082: Deleting consensus metadata
I20260620 09:03:12.913477 16325 catalog_manager.cc:4841] TS bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879): tablet 1846e32cb790484e88e0357abc0a7b3d (table test-table [id=43f199d9f75e49a3bbffc9d55db72024]) successfully deleted
W20260620 09:03:12.991555 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:12.991796 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:12.991835 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:12.995872 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:12.995927 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:12.996025 16532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:13.001283 16532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45177
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:13.002146 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:13.003026 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:13.012180 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:13.012180 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:13.012408 16532 server_base.cc:1034] running on GCE node
W20260620 09:03:13.012494 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:13.012915 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:13.013540 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:13.014703 16532 hybrid_clock.cc:648] HybridClock initialized: now 1781946193014690 us; error 23 us; skew 500 ppm
Jun 20 09:03:13 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:13.017411 16532 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:03:13.018190 16532 webserver.cc:460] Webserver started at http://127.0.98.2:45177/ using document root <none> and password file <none>
I20260620 09:03:13.018716 16532 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:13.018785 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:13.022470 16532 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:13.024362 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:13.025584 16532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:13.025753 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "c17895d23e774900bb261801ee5c0dc3"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "3b1a394fce97fde32940b9a04b78bd74"
server_key_iv: "1ad65d5e762cdec00e4daa5e6f6846dc"
server_key_version: "encryptionkey@0"
I20260620 09:03:13.026340 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:13.045701 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:13.049100 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:13.049309 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:13.049937 16532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:13.050981 16558 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:03:13.054965 16532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:03:13.055035 16532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 09:03:13.055127 16532 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:03:13.056743 16532 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:03:13.056813 16532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260620 09:03:13.057083 16558 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Bootstrap starting.
I20260620 09:03:13.067277 16532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35531
I20260620 09:03:13.067369 16665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35531 every 8 connection(s)
I20260620 09:03:13.068312 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:13.074224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16532
I20260620 09:03:13.074777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34803
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=42761
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:03:13.076004 16558 log.cc:824] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:03:13 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:13.084352 16558 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:03:13.084825 16558 tablet_bootstrap.cc:492] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Bootstrap complete.
I20260620 09:03:13.085028 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.070267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58013 (local address 127.0.98.62:36953)
0620 09:03:13.070438 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.070443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.071120 (+ 677us) server_negotiation.cc:407] Connection header received
0620 09:03:13.072066 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.072073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.072143 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.072266 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:13.073546 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.074501 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.076485 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.076839 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.076976 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.080189 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:13.080195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:13.080199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:13.080243 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:13.082265 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.082873 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.082880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.082884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.082994 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.083384 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.083387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.083390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.083667 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:13.083782 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.084398 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.084700 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:03:13.085733 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.070437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:58013)
0620 09:03:13.070904 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.070966 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.071824 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.072421 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.072429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.072870 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:13.073373 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.073387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.074674 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.074681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.076231 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.076246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.077010 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.077016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.077259 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.077994 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:13.078020 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:13.079978 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:13.082402 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:13.082408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:13.082429 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:13.082757 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:13.083102 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:13.083105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:13.083107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:13.083272 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:13.083790 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:13.083796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:13.084093 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.084319 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.084720 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
I20260620 09:03:13.086203 16558 ts_tablet_manager.cc:1397] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.020s
I20260620 09:03:13.087002 16666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:03:13.087363 16666 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:13.087954 16666 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
I20260620 09:03:13.089016 16558 raft_consensus.cc:348] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:03:13.089535 16558 raft_consensus.cc:729] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17895d23e774900bb261801ee5c0dc3, State: Initialized, Role: FOLLOWER
I20260620 09:03:13.089980 16339 ts_manager.cc:194] Registered new tserver with Master: c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
I20260620 09:03:13.090101 16558 consensus_queue.cc:260] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } } peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } }
I20260620 09:03:13.091280 16558 ts_tablet_manager.cc:1428] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 09:03:13.091280 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58013
I20260620 09:03:13.093856 16666 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
I20260620 09:03:13.097990 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.091720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.0.1:47192)
0620 09:03:13.092013 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.092027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.092105 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.093190 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.093194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.093213 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:13.093315 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.093322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.094950 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.094954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.096284 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.096296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.097582 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.097586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.097684 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.097726 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.097796 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.097868 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260620 09:03:13.098057 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.091973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.0.98.2:35531)
0620 09:03:13.092685 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.092696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.092719 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:03:13.092787 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.092793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.093015 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.093456 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:13.093568 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.094803 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.096501 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.097388 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.097558 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.097706 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.097825 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.097924 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":103,"threads_started":1}
I20260620 09:03:13.099262 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1846e32cb790484e88e0357abc0a7b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47192
I20260620 09:03:13.100211 16673 tablet_replica.cc:330] stopping tablet replica
I20260620 09:03:13.100597 16673 raft_consensus.cc:2229] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:13.100876 16673 raft_consensus.cc:2258] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:13.102005 16673 ts_tablet_manager.cc:1905] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:13.106739 16673 ts_tablet_manager.cc:1918] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:13.106850 16673 log.cc:1196] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/1846e32cb790484e88e0357abc0a7b3d
I20260620 09:03:13.107127 16673 ts_tablet_manager.cc:1939] T 1846e32cb790484e88e0357abc0a7b3d P c17895d23e774900bb261801ee5c0dc3: Deleting consensus metadata
I20260620 09:03:13.109001 16325 catalog_manager.cc:4841] TS c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): tablet 1846e32cb790484e88e0357abc0a7b3d (table test-table [id=43f199d9f75e49a3bbffc9d55db72024]) successfully deleted
W20260620 09:03:13.198567 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:13.198823 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:13.198865 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:13.203006 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:13.203094 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:13.203177 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:13.208357 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42761
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:13.209407 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:13.210376 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:13.219655 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:13.219808 16669 server_base.cc:1034] running on GCE node
W20260620 09:03:13.219838 16679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:13.219691 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:13.220355 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:13.221047 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:13.222215 16669 hybrid_clock.cc:648] HybridClock initialized: now 1781946193222202 us; error 33 us; skew 500 ppm
Jun 20 09:03:13 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:13.224901 16669 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:03:13.225703 16669 webserver.cc:460] Webserver started at http://127.0.98.3:42761/ using document root <none> and password file <none>
I20260620 09:03:13.226210 16669 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:13.226334 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:13.230109 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 09:03:13.232267 16688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:13.233603 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:03:13.233741 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "43257bc3874a4a9d883cf10bdb610f91"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "0d051bedd94de73c96c797e6d11fafff"
server_key_iv: "7c87da4f4a16f9c903dab455123debdc"
server_key_version: "encryptionkey@0"
I20260620 09:03:13.234261 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:13.243242 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:13.246299 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:13.246487 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:13.247027 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:13.247779 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:13.247848 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:13.247920 16669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:13.247969 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:13.256449 16669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34803
I20260620 09:03:13.256524 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34803 every 8 connection(s)
I20260620 09:03:13.257433 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:13.262328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16669
Jun 20 09:03:13 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:13.271117 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.262998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.0.98.62:36953)
0620 09:03:13.263231 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.263235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.263406 (+ 171us) server_negotiation.cc:407] Connection header received
0620 09:03:13.263566 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.263569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.263614 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.263703 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:13.264370 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.265041 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.266013 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.266378 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.266500 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.267932 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:13.267935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:13.267938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:13.267979 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:13.269429 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.269856 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.269858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.269859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.269967 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.270245 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.270247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.270248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.270432 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:13.270642 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.270756 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.270869 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260620 09:03:13.271421 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.259027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50753 (local address 127.0.98.62:36953)
0620 09:03:13.259168 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.259173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.260397 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 09:03:13.261246 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.261248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.261293 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.261366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:13.262622 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.263285 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.264619 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.264826 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.264903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.267533 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:13.267536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:13.267538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:13.267559 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:13.269197 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.269743 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.269748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.269749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.269834 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.270194 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.270198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.270203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.270402 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:13.270488 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.271126 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.271262 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260620 09:03:13.272315 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.259441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:50753)
0620 09:03:13.260254 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.260292 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.261070 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.261525 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.261531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.261850 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:13.262467 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.262487 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.263428 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.263432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.264516 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.264526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.264963 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.264967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.265161 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.265708 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:13.265723 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:13.267353 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:13.269305 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:13.269311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:13.269326 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:13.269647 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:13.269925 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:13.269931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:13.269934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:13.270095 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:13.270517 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:13.270523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:13.270794 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.271047 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.271357 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":507,"threads_started":1}
I20260620 09:03:13.273485 16802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36953
I20260620 09:03:13.273667 16802 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:13.274066 16802 heartbeater.cc:511] Master 127.0.98.62:36953 requested a full tablet report, sending...
I20260620 09:03:13.274951 16339 ts_manager.cc:194] Registered new tserver with Master: 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803)
I20260620 09:03:13.275454 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50753
I20260620 09:03:13.284525 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:13.288169 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.285115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.0.98.62:36953)
0620 09:03:13.285253 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.285256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.285450 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:03:13.285570 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.285572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.285658 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.285740 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:13.285928 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.286446 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.287212 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.287407 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.287492 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.287775 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:13.287898 (+ 123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 09:03:13.288079 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50612: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 09:03:13.288242 16530 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50612: token expired
W20260620 09:03:13.288578 16267 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36953)
I20260620 09:03:13.295109 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.288909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.98.62:36953)
0620 09:03:13.289023 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.289026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.289038 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:03:13.289109 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.289111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.289141 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.289196 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:13.289720 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.290156 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.290857 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.291041 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.291110 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.292224 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:13.292227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:13.292229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:13.292249 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:13.293719 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.294201 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.294205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.294206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.294286 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:13.294529 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:13.294532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:13.294533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:13.294679 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:13.294777 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.294931 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.295009 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:03:13.297389 16339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:13.300554 16339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:13.307497 16600 tablet_service.cc:1467] Processing CreateTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc (DEFAULT_TABLE table=test-table [id=9f40247169db4dd296b2b00fe8c2e526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:13.307497 16461 tablet_service.cc:1467] Processing CreateTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc (DEFAULT_TABLE table=test-table [id=9f40247169db4dd296b2b00fe8c2e526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:13.308087 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca202dc335446fb9eb20a73c1ce5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:13.308087 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca202dc335446fb9eb20a73c1ce5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:13.310864 16558 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Bootstrap starting.
I20260620 09:03:13.310935 16419 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap starting.
I20260620 09:03:13.311381 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.306644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34803 (local address 127.0.0.1:33136)
0620 09:03:13.306758 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.306773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.306849 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.307788 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.307793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.307812 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:13.307893 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.307898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.309191 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.309197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.310182 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.310191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.311022 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.311026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.311111 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.311134 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.311184 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.311254 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:03:13.311682 16558 tablet_bootstrap.cc:654] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:13.311797 16419 tablet_bootstrap.cc:654] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:13.312034 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.306971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.0.98.3:34803)
0620 09:03:13.307365 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.307374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.307391 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:03:13.307458 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.307462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.307619 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.308020 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:13.308181 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.309056 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.310300 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.310884 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.311560 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.311721 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.311777 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.311891 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
I20260620 09:03:13.313154 16558 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: No bootstrap required, opened a new log
I20260620 09:03:13.313220 16558 ts_tablet_manager.cc:1397] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 09:03:13.313741 16419 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: No bootstrap required, opened a new log
I20260620 09:03:13.313829 16419 ts_tablet_manager.cc:1397] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:03:13.313956 16558 raft_consensus.cc:348] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.314070 16558 raft_consensus.cc:374] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:13.314113 16558 raft_consensus.cc:729] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17895d23e774900bb261801ee5c0dc3, State: Initialized, Role: FOLLOWER
I20260620 09:03:13.314245 16558 consensus_queue.cc:260] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.314262 16419 raft_consensus.cc:348] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.314533 16558 ts_tablet_manager.cc:1428] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:03:13.314374 16419 raft_consensus.cc:374] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:13.314651 16419 raft_consensus.cc:729] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fe675fe3a45a39f92b4ab97654082, State: Initialized, Role: FOLLOWER
I20260620 09:03:13.314798 16419 consensus_queue.cc:260] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.315075 16419 ts_tablet_manager.cc:1428] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:03:13.315161 16736 tablet_service.cc:1467] Processing CreateTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc (DEFAULT_TABLE table=test-table [id=9f40247169db4dd296b2b00fe8c2e526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:13.316303 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca202dc335446fb9eb20a73c1ce5bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 09:03:13.319337 16667 tablet.cc:2369] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:13.323248 16814 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Bootstrap starting.
I20260620 09:03:13.325503 16814 tablet_bootstrap.cc:654] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:13.326562 16814 log.cc:824] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:13.328325 16814 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: No bootstrap required, opened a new log
I20260620 09:03:13.328576 16814 ts_tablet_manager.cc:1397] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:03:13.331930 16814 raft_consensus.cc:348] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.332192 16814 raft_consensus.cc:374] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:13.332276 16814 raft_consensus.cc:729] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43257bc3874a4a9d883cf10bdb610f91, State: Initialized, Role: FOLLOWER
I20260620 09:03:13.332808 16814 consensus_queue.cc:260] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.333457 16802 heartbeater.cc:503] Master 127.0.98.62:36953 was elected leader, sending a full tablet report...
I20260620 09:03:13.333714 16814 ts_tablet_manager.cc:1428] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 09:03:13.366549 16528 tablet.cc:2369] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:13.368203 16816 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:13.368434 16816 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.369613 16816 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
I20260620 09:03:13.373318 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.369959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42041 (local address 127.0.98.1:39879)
0620 09:03:13.370090 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.370095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.370111 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:03:13.370173 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.370176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.370238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.370351 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:13.370640 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.371211 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.372262 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.372794 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.372928 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.373093 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.373132 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.373198 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:03:13.373318 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.369853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39879 (local address 127.0.98.3:42041)
0620 09:03:13.369988 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.370004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.370090 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.370369 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.370372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.370389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:13.370518 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.370524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.371330 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.371335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.372131 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.372139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.372933 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.372936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.373033 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.373054 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.373120 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.373187 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":6}
I20260620 09:03:13.373672 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.370126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.98.3:37863)
0620 09:03:13.370564 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.370575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:13.370689 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:13.370971 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:13.370973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:13.370989 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:13.371118 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.371123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.371828 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.371833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.372610 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:13.372619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.373269 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.373272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:13.373370 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.373400 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:03:13.373492 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:03:13.373570 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":155,"threads_started":1}
I20260620 09:03:13.373721 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.370233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37863 (local address 127.0.98.2:35531)
0620 09:03:13.370383 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.370388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.370663 (+ 275us) server_negotiation.cc:407] Connection header received
0620 09:03:13.370812 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.370815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.370864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.370952 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:13.371229 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.371716 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.372738 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.373151 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.373266 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.373323 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.373533 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.373609 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 09:03:13.374272 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca202dc335446fb9eb20a73c1ce5bc" candidate_uuid: "43257bc3874a4a9d883cf10bdb610f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7fe675fe3a45a39f92b4ab97654082" is_pre_election: true
I20260620 09:03:13.374286 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca202dc335446fb9eb20a73c1ce5bc" candidate_uuid: "43257bc3874a4a9d883cf10bdb610f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17895d23e774900bb261801ee5c0dc3" is_pre_election: true
I20260620 09:03:13.374619 16481 raft_consensus.cc:2454] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43257bc3874a4a9d883cf10bdb610f91 in term 0.
I20260620 09:03:13.374617 16620 raft_consensus.cc:2454] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43257bc3874a4a9d883cf10bdb610f91 in term 0.
I20260620 09:03:13.375057 16691 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3; no voters:
I20260620 09:03:13.375285 16816 raft_consensus.cc:2781] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:13.375385 16816 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:13.375445 16816 raft_consensus.cc:3037] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:13.376533 16816 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.376879 16816 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [CANDIDATE]: Term 1 election: Requested vote from peers bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
I20260620 09:03:13.377235 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca202dc335446fb9eb20a73c1ce5bc" candidate_uuid: "43257bc3874a4a9d883cf10bdb610f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7fe675fe3a45a39f92b4ab97654082"
I20260620 09:03:13.377238 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca202dc335446fb9eb20a73c1ce5bc" candidate_uuid: "43257bc3874a4a9d883cf10bdb610f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17895d23e774900bb261801ee5c0dc3"
I20260620 09:03:13.377367 16620 raft_consensus.cc:3037] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:13.377395 16481 raft_consensus.cc:3037] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:13.378664 16620 raft_consensus.cc:2454] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43257bc3874a4a9d883cf10bdb610f91 in term 1.
I20260620 09:03:13.378664 16481 raft_consensus.cc:2454] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43257bc3874a4a9d883cf10bdb610f91 in term 1.
I20260620 09:03:13.378943 16691 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43257bc3874a4a9d883cf10bdb610f91, bf7fe675fe3a45a39f92b4ab97654082; no voters:
I20260620 09:03:13.379101 16816 raft_consensus.cc:2781] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:13.379328 16816 raft_consensus.cc:686] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 1 LEADER]: Becoming Leader. State: Replica: 43257bc3874a4a9d883cf10bdb610f91, State: Running, Role: LEADER
I20260620 09:03:13.379637 16816 consensus_queue.cc:237] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:13.383002 16339 catalog_manager.cc:5495] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "43257bc3874a4a9d883cf10bdb610f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260620 09:03:13.411310 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:13.407878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.0.98.3:34803)
0620 09:03:13.408022 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:13.408026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:13.408059 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:03:13.408125 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:13.408127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:13.408178 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:13.408257 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:13.408497 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.409102 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.409945 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:13.410154 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:13.410242 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:13.410474 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:13.410871 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:13.411109 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:13.411146 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 09:03:13.411194 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:03:13.417124 16481 raft_consensus.cc:1261] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Refusing update from remote peer 43257bc3874a4a9d883cf10bdb610f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:13.417147 16620 raft_consensus.cc:1261] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Refusing update from remote peer 43257bc3874a4a9d883cf10bdb610f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:13.417758 16818 consensus_queue.cc:1035] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:13.417927 16816 consensus_queue.cc:1035] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:13.422786 16821 mvcc.cc:205] Tried to move back new op lower bound from 7298851608229322752 to 7298851608085651456. Current Snapshot: MvccSnapshot[applied={T|T < 7298851608229322752}]
I20260620 09:03:13.424306 16822 mvcc.cc:205] Tried to move back new op lower bound from 7298851608229322752 to 7298851608085651456. Current Snapshot: MvccSnapshot[applied={T|T < 7298851608229322752}]
I20260620 09:03:13.426582 16823 mvcc.cc:205] Tried to move back new op lower bound from 7298851608229322752 to 7298851608085651456. Current Snapshot: MvccSnapshot[applied={T|T < 7298851608229322752 or (T in {7298851608229322752})}]
I20260620 09:03:13.433131 16339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:13.433269 16339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:13.435469 16339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Sending DeleteTablet for 3 replicas of tablet fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:13.436100 16736 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33136
I20260620 09:03:13.436141 16461 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53864
I20260620 09:03:13.436182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16377
I20260620 09:03:13.436981 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca202dc335446fb9eb20a73c1ce5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47192
I20260620 09:03:13.437173 16673 tablet_replica.cc:330] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: stopping tablet replica
I20260620 09:03:13.437326 16673 raft_consensus.cc:2229] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:13.437479 16673 raft_consensus.cc:2258] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:13.438263 16673 ts_tablet_manager.cc:1905] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:13.441463 16673 ts_tablet_manager.cc:1918] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:13.441551 16673 log.cc:1196] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:13.441812 16673 ts_tablet_manager.cc:1939] T fcca202dc335446fb9eb20a73c1ce5bc P c17895d23e774900bb261801ee5c0dc3: Deleting consensus metadata
I20260620 09:03:13.441954 16831 tablet_replica.cc:330] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: stopping tablet replica
I20260620 09:03:13.442318 16831 raft_consensus.cc:2229] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:03:13.442520 16325 catalog_manager.cc:4841] TS c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): tablet fcca202dc335446fb9eb20a73c1ce5bc (table test-table [id=9f40247169db4dd296b2b00fe8c2e526]) successfully deleted
I20260620 09:03:13.442823 16831 raft_consensus.cc:2258] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:03:13.443747 16325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39879 (error 108)
I20260620 09:03:13.444365 16831 ts_tablet_manager.cc:1905] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:03:13.445003 16534 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:03:13.444689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39879 (local address 127.0.0.1:53868)
0620 09:03:13.444818 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:13.444897 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39879: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:03:13.445160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16532
I20260620 09:03:13.447002 16831 ts_tablet_manager.cc:1918] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:13.447093 16831 log.cc:1196] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:13.447417 16831 ts_tablet_manager.cc:1939] T fcca202dc335446fb9eb20a73c1ce5bc P 43257bc3874a4a9d883cf10bdb610f91: Deleting consensus metadata
I20260620 09:03:13.448537 16326 catalog_manager.cc:4841] TS 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): tablet fcca202dc335446fb9eb20a73c1ce5bc (table test-table [id=9f40247169db4dd296b2b00fe8c2e526]) successfully deleted
W20260620 09:03:13.451333 16325 catalog_manager.cc:4584] TS bf7fe675fe3a45a39f92b4ab97654082 (127.0.98.1:39879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcca202dc335446fb9eb20a73c1ce5bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39879: connect: Connection reset by peer (error 104)
I20260620 09:03:13.452142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16669
I20260620 09:03:13.458039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16306
I20260620 09:03:33.465591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--webserver_interface=127.0.98.62
--webserver_port=35505
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:33.578987 16833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:33.579257 16833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:33.579326 16833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:33.583350 16833 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:03:33.583403 16833 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:03:33.583418 16833 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:03:33.583432 16833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:33.583451 16833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:33.583510 16833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:33.583534 16833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:33.583565 16833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:33.588896 16833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:33.589965 16833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:33.591012 16833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:33.599565 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:33.599561 16839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:33.599561 16841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:33.599965 16833 server_base.cc:1034] running on GCE node
I20260620 09:03:33.600625 16833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:33.601683 16833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:33.602850 16833 hybrid_clock.cc:648] HybridClock initialized: now 1781946213602835 us; error 28 us; skew 500 ppm
Jun 20 09:03:33 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:33.605625 16833 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:03:33.606904 16833 webserver.cc:460] Webserver started at http://127.0.98.62:35505/ using document root <none> and password file <none>
I20260620 09:03:33.607579 16833 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:33.607676 16833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:33.611574 16833 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:33.613849 16848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:33.615139 16833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:33.615284 16833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "ef123959f28d4fcda201433145da3260"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "fa45ee1a6df54ef87068ca05aa52d825"
server_key_iv: "429964308ae64066854293e360084f42"
server_key_version: "encryptionkey@0"
I20260620 09:03:33.615845 16833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:33.634500 16833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:33.637938 16833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:33.638157 16833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:33.645517 16833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36953
I20260620 09:03:33.645820 16900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36953 every 8 connection(s)
I20260620 09:03:33.646621 16833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:33.650871 16901 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:03:33.652046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16833
I20260620 09:03:33.652572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39879
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=33997
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:03:33.652803 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap starting.
I20260620 09:03:33.670195 16901 log.cc:824] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:33.680985 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:03:33.681366 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Bootstrap complete.
I20260620 09:03:33.685290 16901 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:33.685647 16901 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef123959f28d4fcda201433145da3260, State: Initialized, Role: FOLLOWER
I20260620 09:03:33.686110 16901 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:33.686229 16901 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:33.686316 16901 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:33.686424 16901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:03:33.688277 16901 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:33.688624 16901 leader_election.cc:302] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef123959f28d4fcda201433145da3260; no voters:
I20260620 09:03:33.688948 16901 leader_election.cc:288] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:03:33.689026 16905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:03:33.689225 16905 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [term 3 LEADER]: Becoming Leader. State: Replica: ef123959f28d4fcda201433145da3260, State: Running, Role: LEADER
I20260620 09:03:33.689507 16905 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } }
I20260620 09:03:33.690172 16901 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:33.690414 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef123959f28d4fcda201433145da3260. Latest consensus state: current_term: 3 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:03:33.690529 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:33.690888 16909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:33.690927 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef123959f28d4fcda201433145da3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef123959f28d4fcda201433145da3260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36953 } } }
I20260620 09:03:33.691138 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:33.695714 16909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:33.696342 16909 catalog_manager.cc:1259] Loaded cluster ID: 97ad8e0d521c4de4a6b9f606d0c8b62c
I20260620 09:03:33.696404 16909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:33.698937 16909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:33.699679 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Loaded TSK: 0
I20260620 09:03:33.699764 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef123959f28d4fcda201433145da3260: Loaded TSK: 1
F20260620 09:03:33.700589 16909 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8bfaf4d9ad google::LogMessage::Fail() at ??:0
@ 0x7f8bfaf5088a google::LogMessage::SendToLog() at ??:0
@ 0x7f8bfaf4d4de google::LogMessage::Flush() at ??:0
@ 0x7f8bfaf4ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8bfdc1118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8bfdc09f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8c058b27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8c0588c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8c0588d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8c058cedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8c058ed82b std::function<>::operator()() at ??:0
@ 0x7f8c0588cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8c0588df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8c0588a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8c058ce640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8bfd19f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8bfd19fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8bfd1a13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8bfd190512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8bfe7d46db start_thread at ??:0
@ 0x7f8bf9a4b61f clone at ??:0
W20260620 09:03:33.799259 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:33.799662 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:33.799749 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:33.804620 16903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:33.804679 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:33.804780 16903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:33.810221 16903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33997
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:33.811318 16903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:33.812354 16903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:33.821862 16949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:33.821862 16948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:33.821856 16951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:33.822351 16903 server_base.cc:1034] running on GCE node
I20260620 09:03:33.822785 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:33.823454 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:33.824628 16903 hybrid_clock.cc:648] HybridClock initialized: now 1781946213824613 us; error 38 us; skew 500 ppm
Jun 20 09:03:33 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:33.827613 16903 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:03:33.828790 16903 webserver.cc:460] Webserver started at http://127.0.98.1:33997/ using document root <none> and password file <none>
I20260620 09:03:33.829483 16903 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:33.829571 16903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:33.834698 16903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:33.837337 16958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:33.838909 16903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:03:33.839031 16903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "bf7fe675fe3a45a39f92b4ab97654082"
format_stamp: "Formatted at 2026-06-20 09:02:51 on dist-test-slave-scsf"
server_key: "73d6070d4542100e16e820a7d467c61b"
server_key_iv: "7849abdd454752d7e5f311ce77495f33"
server_key_version: "encryptionkey@0"
I20260620 09:03:33.839725 16903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:33.853966 16903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:33.857607 16903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:33.857822 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:33.858578 16903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:33.859884 16965 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:03:33.864066 16903 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:03:33.864156 16903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 09:03:33.864238 16903 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:03:33.866485 16903 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:03:33.866549 16903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 09:03:33.866912 16965 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap starting.
I20260620 09:03:33.877784 16903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39879
I20260620 09:03:33.878368 17072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39879 every 8 connection(s)
I20260620 09:03:33.878993 16903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:33.879637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 16903
I20260620 09:03:33.880240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35531
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=45177
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:03:33.887619 16965 log.cc:824] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:33.894680 16965 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:03:33.895084 16965 tablet_bootstrap.cc:492] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Bootstrap complete.
I20260620 09:03:33.896144 16965 ts_tablet_manager.cc:1397] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.012s
I20260620 09:03:33.898474 16965 raft_consensus.cc:348] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:33.898891 16965 raft_consensus.cc:729] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fe675fe3a45a39f92b4ab97654082, State: Initialized, Role: FOLLOWER
I20260620 09:03:33.899361 16965 consensus_queue.cc:260] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:33.900261 16965 ts_tablet_manager.cc:1428] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082: Time spent starting tablet: real 0.004s user 0.006s sys 0.002s
W20260620 09:03:33.947917 17076 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:03:33.881915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:43377)
0620 09:03:33.882515 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:33.882576 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:33.883795 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:33.946976 (+ 63181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":161,"threads_started":1}
W20260620 09:03:33.948518 17073 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36953: Client connection negotiation failed: client connection to 127.0.98.62:36953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:03:33.949235 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:33.948959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:52619)
0620 09:03:33.949096 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:33.949174 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 09:03:33.949863 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:33.949655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:50781)
0620 09:03:33.949760 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:33.949808 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:03:33.950023 17073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:03:34.010804 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:34.011044 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:34.011085 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:34.015074 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:34.015128 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:34.015237 17075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:34.020610 17075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45177
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:34.021591 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:34.022576 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:34.032058 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:34.032095 17075 server_base.cc:1034] running on GCE node
W20260620 09:03:34.032040 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:34.032037 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:34.032732 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:34.033346 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:34.034529 17075 hybrid_clock.cc:648] HybridClock initialized: now 1781946214034514 us; error 27 us; skew 500 ppm
Jun 20 09:03:34 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:34.037122 17075 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:03:34.037981 17075 webserver.cc:460] Webserver started at http://127.0.98.2:45177/ using document root <none> and password file <none>
I20260620 09:03:34.038499 17075 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:34.038558 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:34.042245 17075 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:34.044163 17093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.045395 17075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:34.045495 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "c17895d23e774900bb261801ee5c0dc3"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "3b1a394fce97fde32940b9a04b78bd74"
server_key_iv: "1ad65d5e762cdec00e4daa5e6f6846dc"
server_key_version: "encryptionkey@0"
I20260620 09:03:34.046046 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:34.070688 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:34.073618 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:34.073777 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:34.074347 17075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:34.075157 17075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:34.075203 17075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.075242 17075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:34.075263 17075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.084425 17075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35531
I20260620 09:03:34.084468 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35531 every 8 connection(s)
I20260620 09:03:34.085423 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:34.088136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17075
I20260620 09:03:34.088666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34803
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36953
--webserver_port=42761
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:03:34.089406 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.087484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:44383)
0620 09:03:34.088283 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.088426 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":116,"threads_started":1}
W20260620 09:03:34.089895 17207 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36953: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
I20260620 09:03:34.090488 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.090260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:34635)
0620 09:03:34.090379 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.090427 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 09:03:34.091163 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.090902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:52913)
0620 09:03:34.091036 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.091095 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:03:34.091349 17207 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:03:34.198668 17210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:34.198921 17210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:34.198963 17210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:34.202904 17210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:34.202965 17210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:34.203054 17210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:34.208109 17210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42761
--tserver_master_addrs=127.0.98.62:36953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:34.209170 17210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:34.210165 17210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:34.220108 17218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:34.220088 17215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:34.220155 17210 server_base.cc:1034] running on GCE node
W20260620 09:03:34.220093 17216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:34.220814 17210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:34.221486 17210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:34.222675 17210 hybrid_clock.cc:648] HybridClock initialized: now 1781946214222656 us; error 32 us; skew 500 ppm
Jun 20 09:03:34 dist-test-slave-scsf krb5kdc[15759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:34.225557 17210 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:03:34.226392 17210 webserver.cc:460] Webserver started at http://127.0.98.3:42761/ using document root <none> and password file <none>
I20260620 09:03:34.226907 17210 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:34.226966 17210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:34.230773 17210 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:34.232939 17225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.234170 17210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:34.234297 17210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "43257bc3874a4a9d883cf10bdb610f91"
format_stamp: "Formatted at 2026-06-20 09:02:52 on dist-test-slave-scsf"
server_key: "0d051bedd94de73c96c797e6d11fafff"
server_key_iv: "7c87da4f4a16f9c903dab455123debdc"
server_key_version: "encryptionkey@0"
I20260620 09:03:34.234822 17210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:34.258526 17210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:34.261658 17210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:34.261875 17210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:34.262486 17210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:34.263326 17210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:34.263372 17210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.263439 17210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:34.263481 17210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:34.272698 17210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34803
I20260620 09:03:34.272722 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34803 every 8 connection(s)
I20260620 09:03:34.273837 17210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:34.275466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17210
I20260620 09:03:34.278209 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.276383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:53415)
0620 09:03:34.276954 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.277456 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":159,"threads_started":1}
W20260620 09:03:34.278748 17339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36953: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
I20260620 09:03:34.279340 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.279140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:48249)
0620 09:03:34.279244 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.279287 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 09:03:34.279901 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.279728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:38075)
0620 09:03:34.279818 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.279857 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:03:34.280081 17339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:03:34.951016 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:34.950585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:53195)
0620 09:03:34.950846 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:34.950906 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
I20260620 09:03:35.036851 17344 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:35.037078 17344 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:35.038259 17344 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
Jun 20 09:03:35 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 09:03:35 dist-test-slave-scsf krb5kdc[15759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 09:03:35.052234 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.038574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34803 (local address 127.0.98.1:49611)
0620 09:03:35.039270 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:35.039281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:35.039402 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:35.040667 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:35.040672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:35.041205 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:35.041659 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:35.041673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.043023 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.043029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:35.044054 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:35.044064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.045295 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.045298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:35.045398 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:35.045998 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:35.046013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:35.047670 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:35.049858 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:35.049862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:35.049864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:35.050167 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:35.050514 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:35.050519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:35.050665 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:35.051278 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:35.051284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:35.051514 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 09:03:35.051696 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 09:03:35.052038 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":507,"spinlock_wait_cycles":20864,"thread_start_us":74,"threads_started":1}
I20260620 09:03:35.052248 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.038688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49611 (local address 127.0.98.3:34803)
0620 09:03:35.039017 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:35.039032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:35.039382 (+ 350us) server_negotiation.cc:407] Connection header received
0620 09:03:35.040068 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:35.040073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:35.040367 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:35.040697 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:35.041830 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.042886 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.044174 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.044932 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.045103 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:35.047853 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:35.047857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:35.047867 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:35.047893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:35.049683 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050283 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:35.050288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:35.050300 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:35.050409 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050761 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:35.050765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:35.050767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:35.051156 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:35.051289 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:35.051814 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 09:03:35.052061 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260620 09:03:35.052279 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.038823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41083 (local address 127.0.98.2:35531)
0620 09:03:35.039173 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:35.039186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:35.039237 (+ 51us) server_negotiation.cc:407] Connection header received
0620 09:03:35.039937 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:35.039942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:35.040284 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:35.040629 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:35.041832 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.042938 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.044277 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.044924 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.045103 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:35.047592 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:35.047596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:35.047607 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:35.047646 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:35.049670 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050277 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:35.050283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:35.050287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:35.050418 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050772 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:35.050777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:35.050779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:35.051156 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:35.051289 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:35.051811 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 09:03:35.052091 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
I20260620 09:03:35.052268 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.038567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.98.1:41083)
0620 09:03:35.038722 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:35.038738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:35.038851 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:35.040666 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:35.040672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:35.041214 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:35.041659 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:35.041672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.043106 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.043108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:35.044134 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:35.044148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:35.045041 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:35.045064 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:35.045220 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:35.045813 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:35.045866 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:35.047354 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:35.049792 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:35.049798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:35.049813 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:35.050158 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:35.050521 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:35.050524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:35.050526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:35.050675 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:35.051277 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:35.051283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:35.051514 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 09:03:35.051696 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 09:03:35.052109 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":286,"spinlock_wait_cycles":10112}
W20260620 09:03:35.054945 16962 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
W20260620 09:03:35.055115 16961 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:35.055217 16961 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082; no voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3
I20260620 09:03:35.055464 17344 raft_consensus.cc:2726] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:03:35.092330 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.091906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:37633)
0620 09:03:35.092176 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:35.092234 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260620 09:03:35.281205 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.280802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:49851)
0620 09:03:35.281055 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:35.281113 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260620 09:03:35.952235 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:35.951831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:41793)
0620 09:03:35.952084 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:35.952140 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260620 09:03:36.093516 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:36.093094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:50635)
0620 09:03:36.093331 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:36.093383 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260620 09:03:36.282373 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:36.281886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:51377)
0620 09:03:36.282218 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:36.282273 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":71,"threads_started":1}
I20260620 09:03:36.764278 17353 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:36.764410 17353 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:36.764958 17353 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
W20260620 09:03:36.765774 16962 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
W20260620 09:03:36.765889 16961 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:36.766009 16961 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082; no voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3
I20260620 09:03:36.766186 17353 raft_consensus.cc:2726] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:03:36.953320 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:36.952883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:50951)
0620 09:03:36.953170 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:36.953224 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260620 09:03:37.094839 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:37.094354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:39119)
0620 09:03:37.094680 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:37.094745 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260620 09:03:37.283516 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:37.283087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:54871)
0620 09:03:37.283366 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:37.283423 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260620 09:03:37.954457 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:37.954056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:39767)
0620 09:03:37.954301 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:37.954360 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260620 09:03:38.096344 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:38.095839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:38323)
0620 09:03:38.096191 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:38.096251 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":69,"threads_started":1}
I20260620 09:03:38.284811 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:38.284372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:46683)
0620 09:03:38.284676 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:38.284724 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
I20260620 09:03:38.955935 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:38.955470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:50867)
0620 09:03:38.955778 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:38.955838 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260620 09:03:39.097537 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:39.097130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:33379)
0620 09:03:39.097384 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:39.097443 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260620 09:03:39.286142 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:39.285714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:54983)
0620 09:03:39.285981 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:39.286043 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260620 09:03:39.957127 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:39.956737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:51937)
0620 09:03:39.956970 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:39.957033 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260620 09:03:40.090993 17364 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:40.091148 17364 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:40.091511 17364 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
W20260620 09:03:40.092278 16962 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
W20260620 09:03:40.092372 16961 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:40.092492 16961 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082; no voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3
I20260620 09:03:40.092638 17364 raft_consensus.cc:2726] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:03:40.098649 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:40.098237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:41483)
0620 09:03:40.098510 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:40.098562 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260620 09:03:40.287293 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:40.286910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:58083)
0620 09:03:40.287161 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:40.287212 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260620 09:03:40.958267 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:40.957840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:53663)
0620 09:03:40.958109 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:40.958170 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260620 09:03:41.099790 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:41.099329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:58525)
0620 09:03:41.099646 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:41.099699 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":84,"threads_started":1}
I20260620 09:03:41.288489 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:41.288041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:34235)
0620 09:03:41.288334 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:41.288387 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":67,"threads_started":1}
I20260620 09:03:41.959510 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:41.959078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:34115)
0620 09:03:41.959344 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:41.959405 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260620 09:03:42.101315 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:42.100670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:39437)
0620 09:03:42.101103 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:42.101176 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":77,"threads_started":1}
I20260620 09:03:42.289582 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:42.289170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:52225)
0620 09:03:42.289429 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:42.289489 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260620 09:03:42.551198 17373 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:42.551326 17373 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:42.551761 17373 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
W20260620 09:03:42.552565 16961 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
W20260620 09:03:42.552659 16962 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:42.552726 16962 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082; no voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3
I20260620 09:03:42.552893 17373 raft_consensus.cc:2726] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:03:42.960636 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:42.960184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:57679)
0620 09:03:42.960433 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:42.960537 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260620 09:03:43.102460 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:43.102088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:35503)
0620 09:03:43.102318 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:43.102370 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260620 09:03:43.290702 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:43.290322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:57141)
0620 09:03:43.290561 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:43.290612 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260620 09:03:43.961815 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:43.961429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:41739)
0620 09:03:43.961671 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:43.961723 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260620 09:03:44.103626 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:44.103226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:52945)
0620 09:03:44.103481 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:44.103537 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260620 09:03:44.291787 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:44.291399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:36711)
0620 09:03:44.291643 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:44.291696 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260620 09:03:44.962899 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:44.962495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:34215)
0620 09:03:44.962758 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:44.962808 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260620 09:03:45.105191 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:45.104636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:39333)
0620 09:03:45.105033 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:45.105102 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":99,"threads_started":1}
I20260620 09:03:45.292991 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:45.292625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:57717)
0620 09:03:45.292850 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:45.292902 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260620 09:03:45.778709 17383 raft_consensus.cc:482] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:45.778867 17383 raft_consensus.cc:504] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43257bc3874a4a9d883cf10bdb610f91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34803 } } peers { permanent_uuid: "bf7fe675fe3a45a39f92b4ab97654082" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39879 } } peers { permanent_uuid: "c17895d23e774900bb261801ee5c0dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260620 09:03:45.779347 17383 leader_election.cc:288] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803), c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531)
W20260620 09:03:45.780138 16961 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c17895d23e774900bb261801ee5c0dc3 (127.0.98.2:35531): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
W20260620 09:03:45.780268 16962 leader_election.cc:341] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43257bc3874a4a9d883cf10bdb610f91 (127.0.98.3:34803): Not found: Tablet not found: fcca202dc335446fb9eb20a73c1ce5bc
I20260620 09:03:45.780380 16962 leader_election.cc:302] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf7fe675fe3a45a39f92b4ab97654082; no voters: 43257bc3874a4a9d883cf10bdb610f91, c17895d23e774900bb261801ee5c0dc3
I20260620 09:03:45.780572 17383 raft_consensus.cc:2726] T fcca202dc335446fb9eb20a73c1ce5bc P bf7fe675fe3a45a39f92b4ab97654082 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:03:45.964084 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:45.963679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:47187)
0620 09:03:45.963929 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:45.963980 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260620 09:03:46.106220 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:46.105870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:58669)
0620 09:03:46.106086 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:46.106135 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260620 09:03:46.294070 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:46.293691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:50657)
0620 09:03:46.293925 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:46.293976 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260620 09:03:46.965200 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:46.964827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:38249)
0620 09:03:46.965058 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:46.965109 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260620 09:03:47.107265 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:47.106859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:42267)
0620 09:03:47.107118 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:47.107172 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
I20260620 09:03:47.295284 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:47.294861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:47285)
0620 09:03:47.295132 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:47.295183 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260620 09:03:47.966482 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:47.966084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:47811)
0620 09:03:47.966332 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:47.966387 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260620 09:03:48.108371 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:48.107960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:49825)
0620 09:03:48.108228 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:48.108279 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260620 09:03:48.296329 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:48.295925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.3:51909)
0620 09:03:48.296188 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:48.296236 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260620 09:03:48.967988 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:48.967500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.1:38537)
0620 09:03:48.967830 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:48.967895 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":141,"threads_started":1}
I20260620 09:03:49.109573 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:49.109192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36953 (local address 127.0.98.2:48993)
0620 09:03:49.109432 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:49.109481 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260620 09:03:49.277385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 16903
I20260620 09:03:49.283262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17075
I20260620 09:03:49.290060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17210
2026-06-20T09:03:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59054 ms)
[----------] 30 tests from SecurityITest (241581 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:03:49 dist-test-slave-scsf krb5kdc[17397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:03:49 dist-test-slave-scsf krb5kdc[17397](info): set up 2 sockets
Jun 20 09:03:49 dist-test-slave-scsf krb5kdc[17397](info): commencing operation
krb5kdc: starting...
W20260620 09:03:50.328353 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:03:50 dist-test-slave-scsf krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:03:50.447255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:50.554880 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:50.555119 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:50.555162 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:50.559160 17410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:50.559221 17410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:50.559242 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:50.559257 17410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:50.559300 17410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:50.564531 17410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:50.565503 17410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:50.566502 17410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:50.575029 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:50.575029 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:50.575196 17410 server_base.cc:1034] running on GCE node
W20260620 09:03:50.575150 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:50.575835 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:50.576929 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:50.578111 17410 hybrid_clock.cc:648] HybridClock initialized: now 1781946230578092 us; error 31 us; skew 500 ppm
Jun 20 09:03:50 dist-test-slave-scsf krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:50.580724 17410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:03:50.581501 17410 webserver.cc:460] Webserver started at http://127.0.98.62:35787/ using document root <none> and password file <none>
I20260620 09:03:50.582005 17410 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:50.582062 17410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:50.582227 17410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:50.584136 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "52cbe896a82c4b54adb7b8210965f8d3"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "fb1046fb8fefe92fbdd56e5add5b04cd"
server_key_iv: "6d010b3288e2004fc713829706963887"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.584686 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "52cbe896a82c4b54adb7b8210965f8d3"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "fb1046fb8fefe92fbdd56e5add5b04cd"
server_key_iv: "6d010b3288e2004fc713829706963887"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.588043 17410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:03:50.590374 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:50.591528 17410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:50.591624 17410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "52cbe896a82c4b54adb7b8210965f8d3"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "fb1046fb8fefe92fbdd56e5add5b04cd"
server_key_iv: "6d010b3288e2004fc713829706963887"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.591703 17410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:50.609469 17410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:50.612147 17410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:50.612289 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:50.619009 17410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44219
I20260620 09:03:50.619032 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44219 every 8 connection(s)
I20260620 09:03:50.619901 17410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:50.622540 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:50.622977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17410
I20260620 09:03:50.623113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:50.623422 392 external_mini_cluster.cc:1428] Setting key d13a6cd1a5c5c30597ff4470f7712ee7
I20260620 09:03:50.630066 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: Bootstrap starting.
I20260620 09:03:50.632436 17478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: Neither blocks nor log segments found. Creating new log.
Jun 20 09:03:50 dist-test-slave-scsf krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:50.633363 17478 log.cc:824] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:50.635427 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: No bootstrap required, opened a new log
I20260620 09:03:50.637910 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:50.624851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.0.98.62:44219)
0620 09:03:50.625525 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:50.625543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:50.625585 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:03:50.626398 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:50.626420 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:50.626850 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:50.627221 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:50.627909 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.629417 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.630263 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.630732 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.630898 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:50.633608 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:50.633613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:50.633633 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:50.633660 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:50.635239 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:50.635720 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:50.635725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:50.635730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:50.635833 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:50.636117 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:50.636120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:50.636122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:50.636534 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:50.636699 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:50.636934 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 09:03:50.637202 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":121,"threads_started":1}
I20260620 09:03:50.639278 17478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260620 09:03:50.639509 17478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:50.639581 17478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52cbe896a82c4b54adb7b8210965f8d3, State: Initialized, Role: FOLLOWER
I20260620 09:03:50.640040 17478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260620 09:03:50.640202 17478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:50.640267 17478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:50.640363 17478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:50.641235 17478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260620 09:03:50.641573 17478 leader_election.cc:302] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52cbe896a82c4b54adb7b8210965f8d3; no voters:
I20260620 09:03:50.641907 17478 leader_election.cc:288] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:50.642007 17483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:50.642239 17483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [term 1 LEADER]: Becoming Leader. State: Replica: 52cbe896a82c4b54adb7b8210965f8d3, State: Running, Role: LEADER
I20260620 09:03:50.642565 17483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260620 09:03:50.643061 17478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:50.644310 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52cbe896a82c4b54adb7b8210965f8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260620 09:03:50.644323 17485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52cbe896a82c4b54adb7b8210965f8d3. Latest consensus state: current_term: 1 leader_uuid: "52cbe896a82c4b54adb7b8210965f8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cbe896a82c4b54adb7b8210965f8d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260620 09:03:50.644440 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:50.644490 17485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:50.644879 17492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:50.647848 17492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:50.653064 17492 catalog_manager.cc:1347] Generated new cluster ID: c97c6baf2920490d8fdb75f718680aa0
I20260620 09:03:50.653131 17492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:50.675846 17492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:50.677211 17492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:50.691255 17492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: Generated new TSK 0
I20260620 09:03:50.692318 17492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:03:50.727977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:50.836575 17504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:50.836859 17504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:50.836927 17504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:50.840971 17504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:50.841042 17504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:50.841125 17504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:50.846534 17504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:50.847571 17504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:50.848627 17504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:50.858055 17510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:50.858124 17512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:50.858168 17504 server_base.cc:1034] running on GCE node
W20260620 09:03:50.858340 17509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:50.858786 17504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:50.859400 17504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:50.860579 17504 hybrid_clock.cc:648] HybridClock initialized: now 1781946230860541 us; error 47 us; skew 500 ppm
Jun 20 09:03:50 dist-test-slave-scsf krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:50.863164 17504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:03:50.864231 17504 webserver.cc:460] Webserver started at http://127.0.98.1:36773/ using document root <none> and password file <none>
I20260620 09:03:50.864880 17504 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:50.864972 17504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:50.865176 17504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:50.867159 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "f54dca0924e24f5aab7fc9dbae5a9afa"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "b776070a1d4208dac228c2bb1d7b7db6"
server_key_iv: "8fdb8ff0c4a710455115ca895bd9e98a"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.867702 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f54dca0924e24f5aab7fc9dbae5a9afa"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "b776070a1d4208dac228c2bb1d7b7db6"
server_key_iv: "8fdb8ff0c4a710455115ca895bd9e98a"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.871203 17504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:03:50.873566 17519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:50.874776 17504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:50.874923 17504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "f54dca0924e24f5aab7fc9dbae5a9afa"
format_stamp: "Formatted at 2026-06-20 09:03:50 on dist-test-slave-scsf"
server_key: "b776070a1d4208dac228c2bb1d7b7db6"
server_key_iv: "8fdb8ff0c4a710455115ca895bd9e98a"
server_key_version: "encryptionkey@0"
I20260620 09:03:50.875051 17504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:50.893064 17504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:50.895841 17504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:50.896013 17504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:50.896617 17504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:50.897415 17504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:50.897467 17504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:50.897505 17504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:50.897535 17504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:50.905891 17504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43973
I20260620 09:03:50.905915 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43973 every 8 connection(s)
I20260620 09:03:50.906730 17504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:50.914451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17504
I20260620 09:03:50.914592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:50.914912 392 external_mini_cluster.cc:1428] Setting key 9d5c2d20376822f0e802e8913751579c
Jun 20 09:03:50 dist-test-slave-scsf krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:50.920523 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:50.908557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45229 (local address 127.0.98.62:44219)
0620 09:03:50.908699 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:50.908704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:50.909524 (+ 820us) server_negotiation.cc:407] Connection header received
0620 09:03:50.910726 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:50.910729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:50.910774 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:50.910855 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:50.911909 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.912503 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.913610 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.913847 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.913915 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:50.916763 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:50.916771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:50.916773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:50.916794 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:50.918445 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:50.918921 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:50.918928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:50.918933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:50.919022 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:50.919324 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:50.919329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:50.919334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:50.919594 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:50.919698 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:50.920142 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 09:03:50.920285 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:03:50.921243 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:50.908974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.1:45229)
0620 09:03:50.909386 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:50.909425 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:50.910538 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:50.911003 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:50.911009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:50.911349 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:50.911791 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:50.911802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.912621 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.912625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:50.913507 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:50.913515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:50.913961 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:50.913963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:50.914096 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:50.914786 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:50.914825 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:50.916609 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:50.918567 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:50.918572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:50.918583 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:50.918826 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:50.919105 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:50.919107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:50.919108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:50.919239 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:50.919693 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:50.919698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:50.919910 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 09:03:50.920060 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:03:50.920308 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260620 09:03:50.922434 17633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260620 09:03:50.922668 17633 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:50.923132 17633 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260620 09:03:50.924583 17442 ts_manager.cc:194] Registered new tserver with Master: f54dca0924e24f5aab7fc9dbae5a9afa (127.0.98.1:43973)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:03:50.925719 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:03:50.944013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:51.052904 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:51.053146 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:51.053188 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:51.057364 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:51.057418 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:51.057474 17638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:51.062527 17638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:51.063462 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:51.064365 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:51.073515 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:51.073506 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:51.073505 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:51.073734 17638 server_base.cc:1034] running on GCE node
I20260620 09:03:51.074218 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:51.074855 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:51.076035 17638 hybrid_clock.cc:648] HybridClock initialized: now 1781946231076022 us; error 31 us; skew 500 ppm
Jun 20 09:03:51 dist-test-slave-scsf krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:51.078922 17638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:03:51.079847 17638 webserver.cc:460] Webserver started at http://127.0.98.2:40293/ using document root <none> and password file <none>
I20260620 09:03:51.080397 17638 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:51.080503 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:51.080713 17638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:51.082702 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "e86ba11d40c940c083a08f361223a425"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "d329d50d4290300c72d25303aa3b488e"
server_key_iv: "b1da6edc7d412552d9c64c475e84eecb"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.083279 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e86ba11d40c940c083a08f361223a425"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "d329d50d4290300c72d25303aa3b488e"
server_key_iv: "b1da6edc7d412552d9c64c475e84eecb"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.086894 17638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:03:51.089275 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.090476 17638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:03:51.090590 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "e86ba11d40c940c083a08f361223a425"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "d329d50d4290300c72d25303aa3b488e"
server_key_iv: "b1da6edc7d412552d9c64c475e84eecb"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.090696 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:51.110178 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:51.113091 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:51.113294 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:51.113875 17638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:51.114671 17638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:51.114717 17638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.114781 17638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:51.114820 17638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.123414 17638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44217
I20260620 09:03:51.123440 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44217 every 8 connection(s)
I20260620 09:03:51.124327 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:51.130126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17638
I20260620 09:03:51.130245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:51.130510 392 external_mini_cluster.cc:1428] Setting key f903ff2768ba1a2658f87929801162a4
Jun 20 09:03:51 dist-test-slave-scsf krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:51.138679 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.126152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34263 (local address 127.0.98.62:44219)
0620 09:03:51.126283 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.126286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.127020 (+ 734us) server_negotiation.cc:407] Connection header received
0620 09:03:51.128310 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.128313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.128372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.128488 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:51.129631 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.130155 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.131434 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.131691 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.131799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.134545 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:51.134552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:51.134558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:51.134590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:51.136321 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.136910 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.136913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.136915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.137036 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.137511 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.137517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.137522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.137719 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:51.137798 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.138447 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.138555 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:03:51.139873 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.126457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.2:34263)
0620 09:03:51.126876 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.126917 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.128084 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.128662 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.128669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.129028 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:51.129492 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.129505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.130337 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.130340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.131327 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.131337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.131846 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.131850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.132025 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.132624 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:51.132640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:51.134362 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:51.136438 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:51.136442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:51.136492 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:51.136813 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:51.137147 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:51.137153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:51.137158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:51.137379 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:51.137828 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:51.137838 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:51.138190 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.138402 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.138768 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:03:51.141144 17767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260620 09:03:51.141372 17767 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:51.141896 17767 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260620 09:03:51.142879 17442 ts_manager.cc:194] Registered new tserver with Master: e86ba11d40c940c083a08f361223a425 (127.0.98.2:44217)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:03:51.143556 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:03:51.157868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:51.265316 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:51.265558 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:51.265597 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:51.269405 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:51.269451 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:51.269512 17772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:51.274578 17772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:51.275442 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:51.276420 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:51.285581 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:51.285712 17772 server_base.cc:1034] running on GCE node
W20260620 09:03:51.285619 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:51.285919 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:51.286343 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:51.286963 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:51.288179 17772 hybrid_clock.cc:648] HybridClock initialized: now 1781946231288158 us; error 33 us; skew 500 ppm
Jun 20 09:03:51 dist-test-slave-scsf krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:51.290897 17772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:03:51.291764 17772 webserver.cc:460] Webserver started at http://127.0.98.3:42901/ using document root <none> and password file <none>
I20260620 09:03:51.292305 17772 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:51.292389 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:51.292655 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:51.294703 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "6611be4d7e5a4b3aad4a8e54a288335f"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "bd3d70a1d42662a5b6f0dbe57e34fb02"
server_key_iv: "28c39e980c39404a3068d26b11aa9613"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.295258 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6611be4d7e5a4b3aad4a8e54a288335f"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "bd3d70a1d42662a5b6f0dbe57e34fb02"
server_key_iv: "28c39e980c39404a3068d26b11aa9613"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.298904 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:51.301411 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.302675 17772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:51.302809 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "6611be4d7e5a4b3aad4a8e54a288335f"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "bd3d70a1d42662a5b6f0dbe57e34fb02"
server_key_iv: "28c39e980c39404a3068d26b11aa9613"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.302917 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:51.312207 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:51.314913 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:51.315114 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:51.315634 17772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:51.316399 17772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:51.316480 17772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.316555 17772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:51.316586 17772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.324653 17772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41711
I20260620 09:03:51.324709 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41711 every 8 connection(s)
I20260620 09:03:51.325495 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:51.333403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17772
I20260620 09:03:51.333539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:51.333835 392 external_mini_cluster.cc:1428] Setting key 97175a8bfe0c488f9cdaf1cf541ed128
Jun 20 09:03:51 dist-test-slave-scsf krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:51.338490 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.327200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50607 (local address 127.0.98.62:44219)
0620 09:03:51.327332 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.327336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.328095 (+ 759us) server_negotiation.cc:407] Connection header received
0620 09:03:51.329362 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.329366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.329419 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.329535 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:51.330638 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.331163 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.332258 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.332530 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.332643 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.335193 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:51.335197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:51.335200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:51.335223 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:51.336594 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.337070 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.337074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.337076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.337160 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.337475 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.337480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.337482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.337662 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:51.337764 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.338215 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.338327 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:03:51.339316 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.327553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.3:50607)
0620 09:03:51.327953 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.327993 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.329147 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.329704 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.329710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.330035 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:51.330485 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.330498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.331280 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.331283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.332136 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.332145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.332674 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.332678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.332848 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.333403 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:51.333419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:51.335036 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:51.336707 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:51.336712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:51.336726 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:51.336965 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:51.337252 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:51.337255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:51.337257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:51.337384 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:51.337761 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:51.337768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:51.337988 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.338174 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.338464 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260620 09:03:51.340282 17901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260620 09:03:51.340567 17901 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:51.341007 17901 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260620 09:03:51.341958 17442 ts_manager.cc:194] Registered new tserver with Master: 6611be4d7e5a4b3aad4a8e54a288335f (127.0.98.3:41711)
I20260620 09:03:51.342497 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50607
I20260620 09:03:51.347347 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:51.356101 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.349388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.98.62:44219)
0620 09:03:51.349519 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.349523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.349743 (+ 220us) server_negotiation.cc:407] Connection header received
0620 09:03:51.349839 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.349841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.349878 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.349940 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:51.350608 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.351098 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.351892 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.352090 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.352164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.353507 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:51.353510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:51.353512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:51.353533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:51.354726 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.355095 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.355097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.355098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.355166 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:51.355382 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:51.355384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:51.355385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:51.355572 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:51.355684 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.355831 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.355921 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:03:51.358762 17442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:51.361806 17442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:51.376789 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.371050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.98.2:44217)
0620 09:03:51.371362 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.371373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.371391 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:03:51.371462 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.371466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.371653 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.371801 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:51.372027 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373604 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.374997 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.375849 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376286 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.376434 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.376512 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.376612 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":116,"threads_started":1}
I20260620 09:03:51.376791 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.370417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44217 (local address 127.0.0.1:51228)
0620 09:03:51.370668 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.370681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.370765 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.371802 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.371805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.371824 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:51.371934 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.371940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373764 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.373770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.374852 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.374864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.376155 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.376315 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.376353 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.376492 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.376629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":128,"thread_start_us":111,"threads_started":1}
I20260620 09:03:51.376791 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.369712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43973 (local address 127.0.0.1:54954)
0620 09:03:51.370190 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.370238 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.370387 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.371308 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.371315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.371337 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:51.371484 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.371490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373367 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.373370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.375132 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.375139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.376154 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.376523 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.376542 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.376606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.376655 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":584,"thread_start_us":71,"threads_started":1}
I20260620 09:03:51.377601 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.369892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41711 (local address 127.0.0.1:58144)
0620 09:03:51.370438 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.370452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.370597 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.372112 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.372114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.372128 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:51.372194 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.372198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373283 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.373291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.375098 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.375108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.376154 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.377255 (+ 1097us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.377289 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.377368 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.377452 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":325,"thread_start_us":133,"threads_started":1}
I20260620 09:03:51.377668 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.371192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.0.98.3:41711)
0620 09:03:51.371530 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.371539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.371557 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:03:51.371615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.371618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.371780 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.371927 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:51.372309 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373150 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.375265 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.376032 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376142 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.376299 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.377410 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.377500 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":142,"threads_started":1}
I20260620 09:03:51.380074 17701 tablet_service.cc:1467] Processing CreateTablet for tablet 99e2998960db431880c4ca2123ca45b6 (DEFAULT_TABLE table=test-table [id=206dd70b2aa341be93fdecd4f5222bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:51.380074 17835 tablet_service.cc:1467] Processing CreateTablet for tablet 99e2998960db431880c4ca2123ca45b6 (DEFAULT_TABLE table=test-table [id=206dd70b2aa341be93fdecd4f5222bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:51.380652 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.369897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.0.98.1:43973)
0620 09:03:51.370707 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.370721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.370746 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:03:51.370863 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.370869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.371127 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.371631 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:51.372012 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.373247 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.375277 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.375822 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.376281 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.376430 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.377528 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.380529 (+ 3001us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":127,"threads_started":1}
I20260620 09:03:51.381227 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e2998960db431880c4ca2123ca45b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:51.381227 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e2998960db431880c4ca2123ca45b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:51.383039 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 99e2998960db431880c4ca2123ca45b6 (DEFAULT_TABLE table=test-table [id=206dd70b2aa341be93fdecd4f5222bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:51.383847 17567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e2998960db431880c4ca2123ca45b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:51.388425 17918 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Bootstrap starting.
I20260620 09:03:51.388899 17919 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Bootstrap starting.
I20260620 09:03:51.389797 17920 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Bootstrap starting.
I20260620 09:03:51.391175 17919 tablet_bootstrap.cc:654] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:51.391993 17920 tablet_bootstrap.cc:654] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:51.392231 17918 tablet_bootstrap.cc:654] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:51.392508 17919 log.cc:824] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:51.393165 17920 log.cc:824] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:51.393388 17918 log.cc:824] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:51.394862 17919 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: No bootstrap required, opened a new log
I20260620 09:03:51.394994 17920 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: No bootstrap required, opened a new log
I20260620 09:03:51.395061 17919 ts_tablet_manager.cc:1397] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:03:51.395138 17920 ts_tablet_manager.cc:1397] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260620 09:03:51.395354 17918 tablet_bootstrap.cc:492] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: No bootstrap required, opened a new log
I20260620 09:03:51.395525 17918 ts_tablet_manager.cc:1397] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 09:03:51.398375 17918 raft_consensus.cc:348] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.398680 17918 raft_consensus.cc:374] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:51.398742 17918 raft_consensus.cc:729] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6611be4d7e5a4b3aad4a8e54a288335f, State: Initialized, Role: FOLLOWER
I20260620 09:03:51.398986 17920 raft_consensus.cc:348] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.398986 17919 raft_consensus.cc:348] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.399235 17919 raft_consensus.cc:374] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:51.399235 17920 raft_consensus.cc:374] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:51.399292 17920 raft_consensus.cc:729] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54dca0924e24f5aab7fc9dbae5a9afa, State: Initialized, Role: FOLLOWER
I20260620 09:03:51.399292 17919 raft_consensus.cc:729] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86ba11d40c940c083a08f361223a425, State: Initialized, Role: FOLLOWER
I20260620 09:03:51.399240 17918 consensus_queue.cc:260] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.399745 17920 consensus_queue.cc:260] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.399745 17919 consensus_queue.cc:260] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.399942 17901 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
I20260620 09:03:51.400250 17918 ts_tablet_manager.cc:1428] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 09:03:51.400712 17919 ts_tablet_manager.cc:1428] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:03:51.400712 17920 ts_tablet_manager.cc:1428] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 09:03:51.401171 17767 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
I20260620 09:03:51.401279 17633 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
W20260620 09:03:51.407778 17634 tablet.cc:2369] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:51.514377 17925 raft_consensus.cc:482] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:51.514614 17925 raft_consensus.cc:504] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.515846 17925 leader_election.cc:288] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54dca0924e24f5aab7fc9dbae5a9afa (127.0.98.1:43973), 6611be4d7e5a4b3aad4a8e54a288335f (127.0.98.3:41711)
I20260620 09:03:51.520428 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.516144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41711 (local address 127.0.98.2:39105)
0620 09:03:51.516575 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.516586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.516662 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.516887 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.516889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.516902 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:51.517022 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.517027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.518099 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.518102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.519281 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.519292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.519943 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.519948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.520059 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.520083 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.520183 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.520247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":20,"thread_start_us":127,"threads_started":1}
I20260620 09:03:51.520562 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.516250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39105 (local address 127.0.98.3:41711)
0620 09:03:51.516375 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.516380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.516663 (+ 283us) server_negotiation.cc:407] Connection header received
0620 09:03:51.516732 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.516735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.516786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.516872 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:51.517118 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.517927 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.519400 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.519811 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.519892 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.519941 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.520235 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.520351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:03:51.520601 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.516098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43973 (local address 127.0.98.2:40883)
0620 09:03:51.516311 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.516327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:51.516423 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:51.516759 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:51.516761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:51.516774 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:51.516897 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.516903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.518030 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.518034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.518998 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:51.519008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.520081 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.520085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:51.520224 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.520252 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:03:51.520339 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:03:51.520410 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":22}
I20260620 09:03:51.520598 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.516267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40883 (local address 127.0.98.1:43973)
0620 09:03:51.516400 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.516406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.516426 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:03:51.516576 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.516578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.516638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.516735 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:51.517021 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.517912 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.519142 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.519942 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.520057 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.520122 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.520365 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.520472 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:03:51.521230 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e2998960db431880c4ca2123ca45b6" candidate_uuid: "e86ba11d40c940c083a08f361223a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" is_pre_election: true
I20260620 09:03:51.521229 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e2998960db431880c4ca2123ca45b6" candidate_uuid: "e86ba11d40c940c083a08f361223a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" is_pre_election: true
I20260620 09:03:51.521552 17855 raft_consensus.cc:2454] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e86ba11d40c940c083a08f361223a425 in term 0.
I20260620 09:03:51.521550 17587 raft_consensus.cc:2454] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e86ba11d40c940c083a08f361223a425 in term 0.
I20260620 09:03:51.522064 17657 leader_election.cc:302] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6611be4d7e5a4b3aad4a8e54a288335f, e86ba11d40c940c083a08f361223a425, f54dca0924e24f5aab7fc9dbae5a9afa; no voters:
I20260620 09:03:51.522358 17925 raft_consensus.cc:2781] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:51.522430 17925 raft_consensus.cc:482] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:51.522461 17925 raft_consensus.cc:3037] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:51.523478 17925 raft_consensus.cc:504] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.523842 17925 leader_election.cc:288] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [CANDIDATE]: Term 1 election: Requested vote from peers f54dca0924e24f5aab7fc9dbae5a9afa (127.0.98.1:43973), 6611be4d7e5a4b3aad4a8e54a288335f (127.0.98.3:41711)
I20260620 09:03:51.524191 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e2998960db431880c4ca2123ca45b6" candidate_uuid: "e86ba11d40c940c083a08f361223a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6611be4d7e5a4b3aad4a8e54a288335f"
I20260620 09:03:51.524219 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e2998960db431880c4ca2123ca45b6" candidate_uuid: "e86ba11d40c940c083a08f361223a425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa"
I20260620 09:03:51.524317 17855 raft_consensus.cc:3037] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:51.524349 17587 raft_consensus.cc:3037] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:51.525378 17855 raft_consensus.cc:2454] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e86ba11d40c940c083a08f361223a425 in term 1.
I20260620 09:03:51.525569 17587 raft_consensus.cc:2454] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e86ba11d40c940c083a08f361223a425 in term 1.
I20260620 09:03:51.525714 17654 leader_election.cc:302] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6611be4d7e5a4b3aad4a8e54a288335f, e86ba11d40c940c083a08f361223a425; no voters:
I20260620 09:03:51.525925 17925 raft_consensus.cc:2781] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:51.526166 17925 raft_consensus.cc:686] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 1 LEADER]: Becoming Leader. State: Replica: e86ba11d40c940c083a08f361223a425, State: Running, Role: LEADER
I20260620 09:03:51.526520 17925 consensus_queue.cc:237] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } }
I20260620 09:03:51.529978 17440 catalog_manager.cc:5495] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 reported cstate change: term changed from 0 to 1, leader changed from <none> to e86ba11d40c940c083a08f361223a425 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e86ba11d40c940c083a08f361223a425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e86ba11d40c940c083a08f361223a425" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } health_report { overall_health: HEALTHY } } }
I20260620 09:03:51.562752 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.559406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.0.98.2:44217)
0620 09:03:51.559526 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.559529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.559542 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:03:51.559662 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.559665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.559717 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.559807 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:51.559984 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.560537 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.561357 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.561593 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.561688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.561990 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:51.562402 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:51.562494 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.562561 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.562606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:03:51.568755 17587 raft_consensus.cc:1261] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa [term 1 FOLLOWER]: Refusing update from remote peer e86ba11d40c940c083a08f361223a425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:51.568755 17855 raft_consensus.cc:1261] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 1 FOLLOWER]: Refusing update from remote peer e86ba11d40c940c083a08f361223a425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:51.569480 17925 consensus_queue.cc:1035] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6611be4d7e5a4b3aad4a8e54a288335f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:51.569700 17928 consensus_queue.cc:1035] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54dca0924e24f5aab7fc9dbae5a9afa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:51.576133 17931 mvcc.cc:205] Tried to move back new op lower bound from 7298851764497268736 to 7298851764335202304. Current Snapshot: MvccSnapshot[applied={T|T < 7298851764497268736}]
W20260620 09:03:51.576370 17902 tablet.cc:2369] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:51.586807 17440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:51.586982 17440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:51.589299 17440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52cbe896a82c4b54adb7b8210965f8d3: Sending DeleteTablet for 3 replicas of tablet 99e2998960db431880c4ca2123ca45b6
I20260620 09:03:51.590121 17567 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e2998960db431880c4ca2123ca45b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54954
I20260620 09:03:51.590111 17701 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e2998960db431880c4ca2123ca45b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260620 09:03:51.590194 17835 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e2998960db431880c4ca2123ca45b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58144
I20260620 09:03:51.590551 17941 tablet_replica.cc:330] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: stopping tablet replica
I20260620 09:03:51.590786 17943 tablet_replica.cc:330] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: stopping tablet replica
I20260620 09:03:51.590873 17941 raft_consensus.cc:2229] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:03:51.591251 17942 tablet_replica.cc:330] T 99e2998960db431880c4ca2123ca45b6 P f54dca0924e24f5aab7fc9dbae5a9afa: stopping tablet replica
I20260620 09:03:51.591261 17941 raft_consensus.cc:2258] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:51.591288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17504
I20260620 09:03:51.591846 17943 raft_consensus.cc:2229] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:51.592198 17943 raft_consensus.cc:2258] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:51.594491 17943 ts_tablet_manager.cc:1905] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:51.594561 17941 ts_tablet_manager.cc:1905] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:51.597288 17941 ts_tablet_manager.cc:1918] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:51.597288 17943 ts_tablet_manager.cc:1918] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:51.597389 17941 log.cc:1196] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/99e2998960db431880c4ca2123ca45b6
I20260620 09:03:51.597507 17943 log.cc:1196] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/99e2998960db431880c4ca2123ca45b6
I20260620 09:03:51.597714 17941 ts_tablet_manager.cc:1939] T 99e2998960db431880c4ca2123ca45b6 P e86ba11d40c940c083a08f361223a425: Deleting consensus metadata
I20260620 09:03:51.597954 17943 ts_tablet_manager.cc:1939] T 99e2998960db431880c4ca2123ca45b6 P 6611be4d7e5a4b3aad4a8e54a288335f: Deleting consensus metadata
W20260620 09:03:51.600104 17429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43973 (error 108)
I20260620 09:03:51.600308 17426 catalog_manager.cc:4841] TS 6611be4d7e5a4b3aad4a8e54a288335f (127.0.98.3:41711): tablet 99e2998960db431880c4ca2123ca45b6 (table test-table [id=206dd70b2aa341be93fdecd4f5222bdc]) successfully deleted
I20260620 09:03:51.600515 17426 catalog_manager.cc:4841] TS e86ba11d40c940c083a08f361223a425 (127.0.98.2:44217): tablet 99e2998960db431880c4ca2123ca45b6 (table test-table [id=206dd70b2aa341be93fdecd4f5222bdc]) successfully deleted
I20260620 09:03:51.600749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17638
I20260620 09:03:51.602005 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.601739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43973 (local address 127.0.0.1:54956)
0620 09:03:51.601867 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:51.601934 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:03:51.602183 17429 catalog_manager.cc:4584] TS f54dca0924e24f5aab7fc9dbae5a9afa (127.0.98.1:43973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99e2998960db431880c4ca2123ca45b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43973: connect: Connection refused (error 111)
I20260620 09:03:51.605947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17772
I20260620 09:03:51.611261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17410
2026-06-20T09:03:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T09:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:51Z Disabled control of system clock
I20260620 09:03:51.641182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40663
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:51.772022 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:51.772267 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:51.772301 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:51.776182 17948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:51.776233 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:51.776248 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:51.776264 17948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:51.776279 17948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:51.780990 17948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:51.781903 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:51.783207 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:51.791306 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:51.791352 17948 server_base.cc:1034] running on GCE node
W20260620 09:03:51.791323 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:51.791625 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:51.792193 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:51.793251 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:51.794407 17948 hybrid_clock.cc:648] HybridClock initialized: now 1781946231794397 us; error 39 us; skew 500 ppm
I20260620 09:03:51.796238 17948 webserver.cc:460] Webserver started at http://127.0.98.62:42849/ using document root <none> and password file <none>
I20260620 09:03:51.796823 17948 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:51.796882 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:51.797087 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:51.799018 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "04e3f5d52aa34fb392581d32427f49e5"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "7bdd519eba30bdbfe7e12b3401c780c4"
server_key_iv: "c3af7fe36f863cfa22ddf5a93ad9aab5"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.799557 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "04e3f5d52aa34fb392581d32427f49e5"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "7bdd519eba30bdbfe7e12b3401c780c4"
server_key_iv: "c3af7fe36f863cfa22ddf5a93ad9aab5"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.802977 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:03:51.805534 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:51.806758 17948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:03:51.806859 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "04e3f5d52aa34fb392581d32427f49e5"
format_stamp: "Formatted at 2026-06-20 09:03:51 on dist-test-slave-scsf"
server_key: "7bdd519eba30bdbfe7e12b3401c780c4"
server_key_iv: "c3af7fe36f863cfa22ddf5a93ad9aab5"
server_key_version: "encryptionkey@0"
I20260620 09:03:51.806936 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:51.845072 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:51.845752 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:51.845904 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:51.852806 17948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40663
I20260620 09:03:51.852881 18014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40663 every 8 connection(s)
I20260620 09:03:51.853776 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:51.856595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 17948
I20260620 09:03:51.856740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:51.857061 392 external_mini_cluster.cc:1428] Setting key 51f77bb4901a9795cdcb011e2bedaaee
I20260620 09:03:51.857307 18015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:51.863665 18015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: Bootstrap starting.
I20260620 09:03:51.866079 18015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:51.866290 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:51.858464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.98.62:40663)
0620 09:03:51.859118 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:51.859136 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:51.859172 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:03:51.860032 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:51.860051 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:51.860525 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:51.860878 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:51.861293 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.862241 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.863113 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:51.863574 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:51.863777 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:51.864657 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:51.864662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:51.864674 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:51.865041 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:51.865093 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:51.865112 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:51.865218 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:51.865330 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 09:03:51.865580 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":136,"threads_started":1}
I20260620 09:03:51.866888 18015 log.cc:824] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:51.868920 18015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: No bootstrap required, opened a new log
I20260620 09:03:51.872721 18015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260620 09:03:51.872944 18015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:51.873035 18015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e3f5d52aa34fb392581d32427f49e5, State: Initialized, Role: FOLLOWER
I20260620 09:03:51.873467 18015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260620 09:03:51.873618 18015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:51.873699 18015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:51.873795 18015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:51.874640 18015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260620 09:03:51.874960 18015 leader_election.cc:302] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e3f5d52aa34fb392581d32427f49e5; no voters:
I20260620 09:03:51.875247 18015 leader_election.cc:288] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:51.875351 18020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:51.875576 18020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [term 1 LEADER]: Becoming Leader. State: Replica: 04e3f5d52aa34fb392581d32427f49e5, State: Running, Role: LEADER
I20260620 09:03:51.875867 18020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260620 09:03:51.876500 18015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:51.877388 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e3f5d52aa34fb392581d32427f49e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } } }
I20260620 09:03:51.877521 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:51.877673 18022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e3f5d52aa34fb392581d32427f49e5. Latest consensus state: current_term: 1 leader_uuid: "04e3f5d52aa34fb392581d32427f49e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3f5d52aa34fb392581d32427f49e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } } }
I20260620 09:03:51.877766 18022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:51.877934 18029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:51.880620 18029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:51.885541 18029 catalog_manager.cc:1347] Generated new cluster ID: a3dc46e8d15548ceb612914b7fd97295
I20260620 09:03:51.885603 18029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:51.903318 18029 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:51.904263 18029 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:51.908023 18029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: Generated new TSK 0
I20260620 09:03:51.908746 18029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:03:51.920114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--builtin_ntp_servers=127.0.98.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:52.031536 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:52.031823 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:52.031864 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:52.035971 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:52.036029 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:52.036091 18039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:52.040925 18039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:52.042001 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:52.043370 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:52.053033 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:52.053027 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:52.053036 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:52.053404 18039 server_base.cc:1034] running on GCE node
I20260620 09:03:52.053815 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:52.054431 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:52.055590 18039 hybrid_clock.cc:648] HybridClock initialized: now 1781946232055576 us; error 38 us; skew 500 ppm
I20260620 09:03:52.057374 18039 webserver.cc:460] Webserver started at http://127.0.98.1:33711/ using document root <none> and password file <none>
I20260620 09:03:52.057958 18039 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:52.058032 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:52.058259 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:52.060346 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7ae879ec4414649a493240eb04e3b5f"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "86c9ee31cbaeeef4a17d42964ac7cfaa"
server_key_iv: "7f448abc4efd8f27118482839b2b8437"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.060962 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7ae879ec4414649a493240eb04e3b5f"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "86c9ee31cbaeeef4a17d42964ac7cfaa"
server_key_iv: "7f448abc4efd8f27118482839b2b8437"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.064532 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:03:52.066905 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.068056 18039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:52.068181 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "d7ae879ec4414649a493240eb04e3b5f"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "86c9ee31cbaeeef4a17d42964ac7cfaa"
server_key_iv: "7f448abc4efd8f27118482839b2b8437"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.068284 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:52.086331 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:52.087016 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:52.087221 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:52.087807 18039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:52.088778 18039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:52.088841 18039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.088908 18039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:52.088954 18039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.097908 18039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41749
I20260620 09:03:52.097935 18166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41749 every 8 connection(s)
I20260620 09:03:52.098824 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:52.106500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18039
I20260620 09:03:52.106658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:52.107015 392 external_mini_cluster.cc:1428] Setting key ace3c41be184c4de8b5768bc60ede580
I20260620 09:03:52.108013 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.100622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42569 (local address 127.0.98.62:40663)
0620 09:03:52.100747 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.100751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.101791 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 09:03:52.103110 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.103113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.103157 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.103230 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:52.104118 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.104727 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.105841 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.106040 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.106113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.107493 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:52.107497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:52.107499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:52.107512 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:52.107532 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:52.107567 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:52.107648 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.107803 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.107874 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:03:52.108904 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.100997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40663 (local address 127.0.98.1:42569)
0620 09:03:52.101645 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.101687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.102924 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.103389 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.103395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.103476 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:52.103994 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.104007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.104838 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.104841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.105739 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.105748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.106171 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.106185 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.106397 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.107244 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:52.107245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:52.107284 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:52.107344 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:52.107650 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:52.107661 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:52.107671 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.107802 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.108075 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260620 09:03:52.108963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--builtin_ntp_servers=127.0.98.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:52.110152 18167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40663
I20260620 09:03:52.110435 18167 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:52.110984 18167 heartbeater.cc:511] Master 127.0.98.62:40663 requested a full tablet report, sending...
I20260620 09:03:52.112372 17979 ts_manager.cc:194] Registered new tserver with Master: d7ae879ec4414649a493240eb04e3b5f (127.0.98.1:41749)
I20260620 09:03:52.113533 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42569
W20260620 09:03:52.216338 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:52.216684 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:52.216755 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:52.220791 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:52.220880 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:52.220983 18170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:52.226138 18170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:52.227255 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:52.228680 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:52.237954 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:52.237946 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:52.237946 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:52.238186 18170 server_base.cc:1034] running on GCE node
I20260620 09:03:52.238673 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:52.239269 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:52.240492 18170 hybrid_clock.cc:648] HybridClock initialized: now 1781946232240456 us; error 41 us; skew 500 ppm
I20260620 09:03:52.242245 18170 webserver.cc:460] Webserver started at http://127.0.98.2:39027/ using document root <none> and password file <none>
I20260620 09:03:52.242830 18170 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:52.242920 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:52.243140 18170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:52.245234 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ac05d627cd545369ab92639cb3f6b76"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "083b00bc0f98532bb680da1717616609"
server_key_iv: "c0d882d6b4be69af2cfb24da53d2c6df"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.245790 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ac05d627cd545369ab92639cb3f6b76"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "083b00bc0f98532bb680da1717616609"
server_key_iv: "c0d882d6b4be69af2cfb24da53d2c6df"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.249599 18170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:03:52.251981 18184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.253278 18170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:52.253408 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "8ac05d627cd545369ab92639cb3f6b76"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "083b00bc0f98532bb680da1717616609"
server_key_iv: "c0d882d6b4be69af2cfb24da53d2c6df"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.253520 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:52.286906 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:52.287649 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:52.287847 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:52.288430 18170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:52.289443 18170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:52.289513 18170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.289592 18170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:52.289640 18170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.298382 18170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45509
I20260620 09:03:52.298413 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45509 every 8 connection(s)
I20260620 09:03:52.299304 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:52.305801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18170
I20260620 09:03:52.305943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:52.306259 392 external_mini_cluster.cc:1428] Setting key 22112a9625b279019caaf03d3d4b4c23
I20260620 09:03:52.308351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--builtin_ntp_servers=127.0.98.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:52.308745 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.301075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52157 (local address 127.0.98.62:40663)
0620 09:03:52.301203 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.301207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.302320 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 09:03:52.303618 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.303620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.303667 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.303739 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:52.304582 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.305180 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.306738 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.306987 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.307099 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.308221 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:52.308224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:52.308226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:52.308237 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:52.308255 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:52.308263 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:52.308319 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.308533 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.308602 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:03:52.309916 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.301430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40663 (local address 127.0.98.2:52157)
0620 09:03:52.302169 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.302210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.303433 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.303866 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.303872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.303952 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:52.304426 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.304438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.305283 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.305285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.306606 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.306618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.307119 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.307122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.307260 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.307928 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:52.307930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:52.307981 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:52.308057 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:52.308343 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:52.308352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:52.308359 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.308534 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.308835 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":117,"threads_started":1}
I20260620 09:03:52.311028 18298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40663
I20260620 09:03:52.311254 18298 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:52.311740 18298 heartbeater.cc:511] Master 127.0.98.62:40663 requested a full tablet report, sending...
I20260620 09:03:52.312831 17979 ts_manager.cc:194] Registered new tserver with Master: 8ac05d627cd545369ab92639cb3f6b76 (127.0.98.2:45509)
I20260620 09:03:52.313372 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52157
W20260620 09:03:52.418910 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:52.419209 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:52.419284 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:52.423454 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:52.423542 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:52.423635 18301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:52.428514 18301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:52.429705 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:52.431142 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:52.440750 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:52.440889 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:52.441041 18301 server_base.cc:1034] running on GCE node
W20260620 09:03:52.440912 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:52.441430 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:52.442029 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:52.443205 18301 hybrid_clock.cc:648] HybridClock initialized: now 1781946232443175 us; error 50 us; skew 500 ppm
I20260620 09:03:52.445024 18301 webserver.cc:460] Webserver started at http://127.0.98.3:46333/ using document root <none> and password file <none>
I20260620 09:03:52.445600 18301 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:52.445685 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:52.445906 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:52.447885 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "33b6af554e0048eab13eed69371e2214"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "885f60e4db3c844c9b83ca49cb288ce5"
server_key_iv: "dc3f8d74b2f6b83f0835d94930698ab1"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.448427 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33b6af554e0048eab13eed69371e2214"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "885f60e4db3c844c9b83ca49cb288ce5"
server_key_iv: "dc3f8d74b2f6b83f0835d94930698ab1"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.451972 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:03:52.454348 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.455482 18301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:52.455622 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "33b6af554e0048eab13eed69371e2214"
format_stamp: "Formatted at 2026-06-20 09:03:52 on dist-test-slave-scsf"
server_key: "885f60e4db3c844c9b83ca49cb288ce5"
server_key_iv: "dc3f8d74b2f6b83f0835d94930698ab1"
server_key_version: "encryptionkey@0"
I20260620 09:03:52.455729 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:52.475219 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:52.475905 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:52.476099 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:52.476781 18301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:52.477746 18301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:52.477854 18301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.477913 18301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:52.477947 18301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:52.486162 18301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35887
I20260620 09:03:52.486244 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35887 every 8 connection(s)
I20260620 09:03:52.487079 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:52.495157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18301
I20260620 09:03:52.495285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:52.495635 392 external_mini_cluster.cc:1428] Setting key a2754acef116ae66b1a9e063e102a6cf
I20260620 09:03:52.495986 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.488937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52799 (local address 127.0.98.62:40663)
0620 09:03:52.489101 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.489106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.490116 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 09:03:52.491412 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.491415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.491461 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.491535 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:52.492379 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.492979 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.494106 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.494317 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.494394 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.495521 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:52.495524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:52.495526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:52.495538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:52.495558 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:52.495565 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:52.495623 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.495800 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.495853 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:03:52.497054 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.489349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40663 (local address 127.0.98.3:52799)
0620 09:03:52.489963 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.490000 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.491236 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.491691 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.491698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.491792 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:52.492266 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.492278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.493116 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.493119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.493997 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.494007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.494446 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.494449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.494602 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.495218 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:52.495220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:52.495281 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:52.495364 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:52.495649 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:52.495659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:52.495665 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.495802 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.496106 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260620 09:03:52.498162 18429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40663
I20260620 09:03:52.498391 18429 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:52.498904 18429 heartbeater.cc:511] Master 127.0.98.62:40663 requested a full tablet report, sending...
I20260620 09:03:52.499872 17979 ts_manager.cc:194] Registered new tserver with Master: 33b6af554e0048eab13eed69371e2214 (127.0.98.3:35887)
I20260620 09:03:52.500433 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52799
I20260620 09:03:52.509671 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:52.515101 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.511697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.98.62:40663)
0620 09:03:52.511844 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.511849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.511994 (+ 145us) server_negotiation.cc:407] Connection header received
0620 09:03:52.512087 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.512089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.512129 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.512210 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:52.512483 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.512977 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.513757 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.513955 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.514028 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.514706 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:52.514709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:52.514711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:52.514721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:52.514739 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:52.514747 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:52.514803 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.514877 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.514922 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:03:52.517717 17979 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:52.520737 17979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:52.534739 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.529545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.98.2:45509)
0620 09:03:52.530369 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.530385 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.530409 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:03:52.530482 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.530487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.530690 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.530836 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:52.531113 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.532157 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.533455 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534066 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.534196 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.534392 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.534479 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.534575 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":138,"threads_started":1}
I20260620 09:03:52.534894 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.528728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35887 (local address 127.0.0.1:35392)
0620 09:03:52.529188 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.529234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.529357 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.530178 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.530185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.530209 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:52.530369 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.530379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.532009 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.532012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.533347 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.533358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534432 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.534435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.534536 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.534557 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.534617 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.534703 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":84,"thread_start_us":74,"threads_started":1}
I20260620 09:03:52.534989 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.528837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.0.98.3:35887)
0620 09:03:52.529651 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.529664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.529694 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:03:52.529768 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.529773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.529988 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.530550 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:52.530695 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.531900 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.533492 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534288 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.534478 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.534664 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.534710 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.534798 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":161,"threads_started":1}
I20260620 09:03:52.534735 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.528816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45509 (local address 127.0.0.1:40758)
0620 09:03:52.529974 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.529987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.530066 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.530884 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.530890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.530908 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:52.530996 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.531001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.532301 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.532306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.533337 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.533346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534194 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.534198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.534300 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.534335 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.534449 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.534569 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":27,"thread_start_us":104,"threads_started":1}
I20260620 09:03:52.535338 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.529446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41749 (local address 127.0.0.1:36424)
0620 09:03:52.530029 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.530046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.530127 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.530702 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.530705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.530723 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:52.530817 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.530824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.531844 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.531849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.533348 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.533358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534432 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.534435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.534924 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.535021 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.535142 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.535214 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":49,"thread_start_us":98,"threads_started":1}
I20260620 09:03:52.535986 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.529604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.0.98.1:41749)
0620 09:03:52.530316 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.530324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.530342 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:03:52.530400 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.530403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.530536 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.530728 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:52.530946 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.531727 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.533492 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.534288 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.535502 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.535692 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.535742 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.535836 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":128,"threads_started":1}
I20260620 09:03:52.538097 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 (DEFAULT_TABLE table=test-table [id=c38bf04f50814ff3a627a8f515106975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:52.538097 18363 tablet_service.cc:1467] Processing CreateTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 (DEFAULT_TABLE table=test-table [id=c38bf04f50814ff3a627a8f515106975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:52.538414 18101 tablet_service.cc:1467] Processing CreateTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 (DEFAULT_TABLE table=test-table [id=c38bf04f50814ff3a627a8f515106975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:52.539228 18363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972d26b4a44a47f2a92c35883abda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:52.539228 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972d26b4a44a47f2a92c35883abda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:52.539326 18101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972d26b4a44a47f2a92c35883abda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:52.545220 18446 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Bootstrap starting.
I20260620 09:03:52.546654 18447 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Bootstrap starting.
I20260620 09:03:52.547374 18446 tablet_bootstrap.cc:654] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:52.548038 18446 log.cc:824] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:52.548399 18447 tablet_bootstrap.cc:654] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:52.548702 18448 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Bootstrap starting.
I20260620 09:03:52.549232 18447 log.cc:824] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:52.550170 18446 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: No bootstrap required, opened a new log
I20260620 09:03:52.550348 18446 ts_tablet_manager.cc:1397] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:03:52.550560 18448 tablet_bootstrap.cc:654] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:52.551162 18447 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: No bootstrap required, opened a new log
I20260620 09:03:52.551237 18448 log.cc:824] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:52.551342 18447 ts_tablet_manager.cc:1397] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 09:03:52.552874 18448 tablet_bootstrap.cc:492] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: No bootstrap required, opened a new log
I20260620 09:03:52.553059 18448 ts_tablet_manager.cc:1397] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 09:03:52.554749 18446 raft_consensus.cc:348] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.555022 18446 raft_consensus.cc:374] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:52.555079 18446 raft_consensus.cc:729] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ae879ec4414649a493240eb04e3b5f, State: Initialized, Role: FOLLOWER
I20260620 09:03:52.555536 18446 consensus_queue.cc:260] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.555464 18447 raft_consensus.cc:348] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.555701 18447 raft_consensus.cc:374] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:52.555758 18447 raft_consensus.cc:729] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac05d627cd545369ab92639cb3f6b76, State: Initialized, Role: FOLLOWER
I20260620 09:03:52.556180 18447 consensus_queue.cc:260] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.556208 18448 raft_consensus.cc:348] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.556433 18448 raft_consensus.cc:374] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:52.556504 18448 raft_consensus.cc:729] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b6af554e0048eab13eed69371e2214, State: Initialized, Role: FOLLOWER
I20260620 09:03:52.556589 18446 ts_tablet_manager.cc:1428] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20260620 09:03:52.556937 18448 consensus_queue.cc:260] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.557116 18447 ts_tablet_manager.cc:1428] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 09:03:52.557108 18167 heartbeater.cc:503] Master 127.0.98.62:40663 was elected leader, sending a full tablet report...
I20260620 09:03:52.557331 18298 heartbeater.cc:503] Master 127.0.98.62:40663 was elected leader, sending a full tablet report...
I20260620 09:03:52.558455 18429 heartbeater.cc:503] Master 127.0.98.62:40663 was elected leader, sending a full tablet report...
I20260620 09:03:52.558756 18448 ts_tablet_manager.cc:1428] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 09:03:52.573338 18452 raft_consensus.cc:482] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:52.573505 18452 raft_consensus.cc:504] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.574450 18452 leader_election.cc:288] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33b6af554e0048eab13eed69371e2214 (127.0.98.3:35887), 8ac05d627cd545369ab92639cb3f6b76 (127.0.98.2:45509)
I20260620 09:03:52.579174 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.574647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35887 (local address 127.0.98.1:51271)
0620 09:03:52.575057 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.575067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.575174 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.575407 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.575409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.575427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:52.575515 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.575519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.576419 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.576422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.577818 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.577829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.578741 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.578745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.578855 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.578874 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.578934 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.578986 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":201,"thread_start_us":88,"threads_started":1}
I20260620 09:03:52.579180 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.574767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51271 (local address 127.0.98.3:35887)
0620 09:03:52.574910 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.574915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.575135 (+ 220us) server_negotiation.cc:407] Connection header received
0620 09:03:52.575277 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.575280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.575317 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.575373 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:52.575602 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.576332 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.578071 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.578635 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.578740 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.578802 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.578965 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.579051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:03:52.579216 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.574647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45509 (local address 127.0.98.1:34909)
0620 09:03:52.574783 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.574801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:52.574892 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:52.575179 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:52.575182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:52.575197 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:52.575323 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.575328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.576321 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.576323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.577810 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:52.577819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.578826 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.578830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:52.578939 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.578963 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:03:52.579038 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:03:52.579102 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":58}
I20260620 09:03:52.579269 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.574764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34909 (local address 127.0.98.2:45509)
0620 09:03:52.574878 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.574881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.574895 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:03:52.575022 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.575025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.575067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.575162 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:52.575451 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.576231 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.577918 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.578544 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.578674 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.578740 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.579074 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.579149 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:03:52.579859 18252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972d26b4a44a47f2a92c35883abda4e6" candidate_uuid: "d7ae879ec4414649a493240eb04e3b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac05d627cd545369ab92639cb3f6b76" is_pre_election: true
I20260620 09:03:52.579962 18383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972d26b4a44a47f2a92c35883abda4e6" candidate_uuid: "d7ae879ec4414649a493240eb04e3b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b6af554e0048eab13eed69371e2214" is_pre_election: true
I20260620 09:03:52.580174 18252 raft_consensus.cc:2454] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ae879ec4414649a493240eb04e3b5f in term 0.
I20260620 09:03:52.580252 18383 raft_consensus.cc:2454] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ae879ec4414649a493240eb04e3b5f in term 0.
I20260620 09:03:52.580695 18056 leader_election.cc:302] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33b6af554e0048eab13eed69371e2214, 8ac05d627cd545369ab92639cb3f6b76, d7ae879ec4414649a493240eb04e3b5f; no voters:
I20260620 09:03:52.580971 18452 raft_consensus.cc:2781] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:52.581041 18452 raft_consensus.cc:482] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:52.581089 18452 raft_consensus.cc:3037] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:52.582077 18452 raft_consensus.cc:504] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.582433 18452 leader_election.cc:288] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [CANDIDATE]: Term 1 election: Requested vote from peers 33b6af554e0048eab13eed69371e2214 (127.0.98.3:35887), 8ac05d627cd545369ab92639cb3f6b76 (127.0.98.2:45509)
I20260620 09:03:52.582722 18383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972d26b4a44a47f2a92c35883abda4e6" candidate_uuid: "d7ae879ec4414649a493240eb04e3b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b6af554e0048eab13eed69371e2214"
I20260620 09:03:52.582760 18252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972d26b4a44a47f2a92c35883abda4e6" candidate_uuid: "d7ae879ec4414649a493240eb04e3b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac05d627cd545369ab92639cb3f6b76"
I20260620 09:03:52.582847 18383 raft_consensus.cc:3037] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:52.582855 18252 raft_consensus.cc:3037] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:52.584084 18252 raft_consensus.cc:2454] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ae879ec4414649a493240eb04e3b5f in term 1.
I20260620 09:03:52.584092 18383 raft_consensus.cc:2454] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ae879ec4414649a493240eb04e3b5f in term 1.
I20260620 09:03:52.584374 18056 leader_election.cc:302] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac05d627cd545369ab92639cb3f6b76, d7ae879ec4414649a493240eb04e3b5f; no voters:
I20260620 09:03:52.584592 18452 raft_consensus.cc:2781] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:52.584772 18452 raft_consensus.cc:686] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 1 LEADER]: Becoming Leader. State: Replica: d7ae879ec4414649a493240eb04e3b5f, State: Running, Role: LEADER
I20260620 09:03:52.585068 18452 consensus_queue.cc:237] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } }
I20260620 09:03:52.587963 17979 catalog_manager.cc:5495] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ae879ec4414649a493240eb04e3b5f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d7ae879ec4414649a493240eb04e3b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ae879ec4414649a493240eb04e3b5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41749 } health_report { overall_health: HEALTHY } } }
I20260620 09:03:52.599113 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.595610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.0.98.1:41749)
0620 09:03:52.595736 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:52.595740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:52.595753 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:03:52.595856 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:52.595859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:52.595910 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:52.595999 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:52.596195 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.596741 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.597600 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:52.597817 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:52.597907 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:52.598148 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:52.598754 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:52.598862 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:52.598934 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:03:52.598980 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 09:03:52.599942 18168 tablet.cc:2369] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:52.604911 18252 raft_consensus.cc:1261] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 1 FOLLOWER]: Refusing update from remote peer d7ae879ec4414649a493240eb04e3b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:52.604911 18383 raft_consensus.cc:1261] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 1 FOLLOWER]: Refusing update from remote peer d7ae879ec4414649a493240eb04e3b5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:52.605612 18456 consensus_queue.cc:1035] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b6af554e0048eab13eed69371e2214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:52.605765 18452 consensus_queue.cc:1035] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac05d627cd545369ab92639cb3f6b76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:52.613582 18460 mvcc.cc:205] Tried to move back new op lower bound from 7298851768742219776 to 7298851768671121408. Current Snapshot: MvccSnapshot[applied={T|T < 7298851768742219776}]
I20260620 09:03:52.614342 18461 mvcc.cc:205] Tried to move back new op lower bound from 7298851768742219776 to 7298851768671121408. Current Snapshot: MvccSnapshot[applied={T|T < 7298851768742219776}]
I20260620 09:03:52.621378 17977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:52.621520 17977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:52.623745 17977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04e3f5d52aa34fb392581d32427f49e5: Sending DeleteTablet for 3 replicas of tablet 972d26b4a44a47f2a92c35883abda4e6
I20260620 09:03:52.624545 18363 tablet_service.cc:1514] Processing DeleteTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:52 UTC) from {username='slave'} at 127.0.0.1:35392
I20260620 09:03:52.624627 18101 tablet_service.cc:1514] Processing DeleteTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:52 UTC) from {username='slave'} at 127.0.0.1:36424
I20260620 09:03:52.624703 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet 972d26b4a44a47f2a92c35883abda4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:52 UTC) from {username='slave'} at 127.0.0.1:40758
I20260620 09:03:52.625048 18469 tablet_replica.cc:330] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f: stopping tablet replica
I20260620 09:03:52.625171 18471 tablet_replica.cc:330] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: stopping tablet replica
I20260620 09:03:52.625346 18469 raft_consensus.cc:2229] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:03:52.625387 18471 raft_consensus.cc:2229] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:52.625671 18471 raft_consensus.cc:2258] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:52.625711 18469 raft_consensus.cc:2258] T 972d26b4a44a47f2a92c35883abda4e6 P d7ae879ec4414649a493240eb04e3b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:52.625938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18039
I20260620 09:03:52.626041 18470 tablet_replica.cc:330] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: stopping tablet replica
I20260620 09:03:52.626320 18470 raft_consensus.cc:2229] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:52.626623 18470 raft_consensus.cc:2258] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:52.628072 18470 ts_tablet_manager.cc:1905] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:52.630548 18470 ts_tablet_manager.cc:1918] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:52.630656 18470 log.cc:1196] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/972d26b4a44a47f2a92c35883abda4e6
I20260620 09:03:52.630970 18470 ts_tablet_manager.cc:1939] T 972d26b4a44a47f2a92c35883abda4e6 P 33b6af554e0048eab13eed69371e2214: Deleting consensus metadata
I20260620 09:03:52.631313 18471 ts_tablet_manager.cc:1905] T 972d26b4a44a47f2a92c35883abda4e6 P 8ac05d627cd545369ab92639cb3f6b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:52.633230 17966 catalog_manager.cc:4841] TS 33b6af554e0048eab13eed69371e2214 (127.0.98.3:35887): tablet 972d26b4a44a47f2a92c35883abda4e6 (table test-table [id=c38bf04f50814ff3a627a8f515106975]) successfully deleted
I20260620 09:03:52.633772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18170
W20260620 09:03:52.633760 17965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41749 (error 108)
I20260620 09:03:52.636209 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.635843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41749 (local address 127.0.0.1:36452)
0620 09:03:52.636006 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.636092 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 09:03:52.636376 17965 catalog_manager.cc:4584] TS d7ae879ec4414649a493240eb04e3b5f (127.0.98.1:41749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 972d26b4a44a47f2a92c35883abda4e6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41749: connect: Connection refused (error 111)
I20260620 09:03:52.639695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18301
I20260620 09:03:52.640662 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:52.639861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45509 (local address 127.0.0.1:40762)
0620 09:03:52.640487 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:52.640557 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260620 09:03:52.645376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 17948
2026-06-20T09:03:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:03:52 dist-test-slave-scsf krb5kdc[18475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:03:52 dist-test-slave-scsf krb5kdc[18475](info): set up 2 sockets
Jun 20 09:03:52 dist-test-slave-scsf krb5kdc[18475](info): commencing operation
krb5kdc: starting...
W20260620 09:03:53.700302 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:03:53 dist-test-slave-scsf krb5kdc[18475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:03:53.816983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:53.923943 18488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:53.924235 18488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:53.924311 18488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:53.928278 18488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:53.928362 18488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:53.928401 18488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:53.928434 18488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:53.928490 18488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:53.933815 18488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:53.934798 18488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:53.935822 18488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:03:53.944310 18488 server_base.cc:1034] running on GCE node
W20260620 09:03:53.944317 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:53.944298 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:53.944298 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:53.945192 18488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:53.946509 18488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:53.947697 18488 hybrid_clock.cc:648] HybridClock initialized: now 1781946233947681 us; error 29 us; skew 500 ppm
Jun 20 09:03:53 dist-test-slave-scsf krb5kdc[18475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:53.950393 18488 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:03:53.951256 18488 webserver.cc:460] Webserver started at http://127.0.98.62:35033/ using document root <none> and password file <none>
I20260620 09:03:53.951802 18488 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:53.951890 18488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:53.952111 18488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:53.954121 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "a648ce36af9b4809b1ac7d6da2c7ef96"
format_stamp: "Formatted at 2026-06-20 09:03:53 on dist-test-slave-scsf"
server_key: "0dc95b4a1724da224129927fec80e01c"
server_key_iv: "6b45b7604568526cf0e8e3ab5a838f6c"
server_key_version: "encryptionkey@0"
I20260620 09:03:53.954674 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "a648ce36af9b4809b1ac7d6da2c7ef96"
format_stamp: "Formatted at 2026-06-20 09:03:53 on dist-test-slave-scsf"
server_key: "0dc95b4a1724da224129927fec80e01c"
server_key_iv: "6b45b7604568526cf0e8e3ab5a838f6c"
server_key_version: "encryptionkey@0"
I20260620 09:03:53.958217 18488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:53.960492 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:53.961694 18488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:03:53.961826 18488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "a648ce36af9b4809b1ac7d6da2c7ef96"
format_stamp: "Formatted at 2026-06-20 09:03:53 on dist-test-slave-scsf"
server_key: "0dc95b4a1724da224129927fec80e01c"
server_key_iv: "6b45b7604568526cf0e8e3ab5a838f6c"
server_key_version: "encryptionkey@0"
I20260620 09:03:53.961927 18488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:53.975011 18488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:53.978025 18488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:53.978227 18488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:53.985165 18488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36771
I20260620 09:03:53.985193 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36771 every 8 connection(s)
I20260620 09:03:53.986042 18488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:53.988654 18556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:53.992497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18488
I20260620 09:03:53.992642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:53.992933 392 external_mini_cluster.cc:1428] Setting key 27e371603d0ef0086b03b855c6aaca36
I20260620 09:03:53.994401 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: Bootstrap starting.
I20260620 09:03:53.996579 18556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:53.997292 18556 log.cc:824] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:53.999025 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: No bootstrap required, opened a new log
Jun 20 09:03:53 dist-test-slave-scsf krb5kdc[18475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:54.002763 18556 raft_consensus.cc:348] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } }
I20260620 09:03:54.003050 18556 raft_consensus.cc:374] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:54.003129 18556 raft_consensus.cc:729] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648ce36af9b4809b1ac7d6da2c7ef96, State: Initialized, Role: FOLLOWER
I20260620 09:03:54.003590 18556 consensus_queue.cc:260] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } }
I20260620 09:03:54.003736 18556 raft_consensus.cc:388] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:54.003814 18556 raft_consensus.cc:482] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:54.003929 18556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:54.004972 18556 raft_consensus.cc:504] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } }
I20260620 09:03:54.005313 18556 leader_election.cc:302] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a648ce36af9b4809b1ac7d6da2c7ef96; no voters:
I20260620 09:03:54.005662 18556 leader_election.cc:288] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:54.005834 18561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:54.006026 18561 raft_consensus.cc:686] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [term 1 LEADER]: Becoming Leader. State: Replica: a648ce36af9b4809b1ac7d6da2c7ef96, State: Running, Role: LEADER
I20260620 09:03:54.006349 18561 consensus_queue.cc:237] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } }
I20260620 09:03:54.006774 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:53.994322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.0.98.62:36771)
0620 09:03:53.994724 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:53.994738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:53.994772 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:03:53.995373 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:53.995387 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:53.995703 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:53.996037 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:53.996932 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:53.997948 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:53.998824 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:53.999328 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:53.999485 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.001939 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:54.001944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:54.001955 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:54.001979 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:54.003684 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.004237 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.004242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.004246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.004349 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.004656 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.004658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.004659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.005008 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:54.005212 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.005488 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.005809 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260620 09:03:54.006960 18556 sys_catalog.cc:564] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:54.007464 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a648ce36af9b4809b1ac7d6da2c7ef96. Latest consensus state: current_term: 1 leader_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } } }
I20260620 09:03:54.007575 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:54.008088 18563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648ce36af9b4809b1ac7d6da2c7ef96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36771 } } }
I20260620 09:03:54.008248 18563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:54.008893 18566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:54.012022 18566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:54.017146 18566 catalog_manager.cc:1347] Generated new cluster ID: c79ffc0c6bd445748525a25631139921
I20260620 09:03:54.017211 18566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:54.032337 18566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:54.033471 18566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:54.038854 18566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: Generated new TSK 0
I20260620 09:03:54.039556 18566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:03:54.075433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:54.217178 18582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:54.217427 18582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:54.217469 18582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:54.221350 18582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:54.221398 18582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:54.221453 18582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:54.226387 18582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:54.227326 18582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:54.228308 18582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:54.237450 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:54.237463 18590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:54.237473 18587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:54.237814 18582 server_base.cc:1034] running on GCE node
I20260620 09:03:54.238149 18582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:54.238714 18582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:54.239918 18582 hybrid_clock.cc:648] HybridClock initialized: now 1781946234239888 us; error 48 us; skew 500 ppm
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:54.242686 18582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:03:54.243477 18582 webserver.cc:460] Webserver started at http://127.0.98.1:37317/ using document root <none> and password file <none>
I20260620 09:03:54.244014 18582 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:54.244100 18582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:54.244310 18582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:54.246280 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff07728b273e4f8a8c0f67ca19c21791"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "9006a53bbaea7c109cabb88183909f89"
server_key_iv: "52e3528b701957f33fd53f54d9d80fd8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.246831 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff07728b273e4f8a8c0f67ca19c21791"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "9006a53bbaea7c109cabb88183909f89"
server_key_iv: "52e3528b701957f33fd53f54d9d80fd8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.250380 18582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:54.252669 18597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.253820 18582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:54.253947 18582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "ff07728b273e4f8a8c0f67ca19c21791"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "9006a53bbaea7c109cabb88183909f89"
server_key_iv: "52e3528b701957f33fd53f54d9d80fd8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.254048 18582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:54.271965 18582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:54.274736 18582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:54.274931 18582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:54.275504 18582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:54.276298 18582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:54.276365 18582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.276430 18582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:54.276497 18582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.285192 18582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34189
I20260620 09:03:54.285220 18710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34189 every 8 connection(s)
I20260620 09:03:54.286053 18582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:54.292043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18582
I20260620 09:03:54.292212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:54.292570 392 external_mini_cluster.cc:1428] Setting key ba2c8f1190c0563ab68192aba9bab5a3
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:54.299571 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.287758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52745 (local address 127.0.98.62:36771)
0620 09:03:54.287894 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.287898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.288700 (+ 802us) server_negotiation.cc:407] Connection header received
0620 09:03:54.289946 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.289950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.290008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.290108 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:54.291251 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.291793 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.293051 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.293344 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.293446 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.295869 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:54.295873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:54.295876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:54.295918 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:54.297587 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.298062 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.298067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.298068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.298156 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.298496 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.298501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.298503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.298694 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:54.298801 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.299275 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.299397 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:03:54.300349 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.288088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36771 (local address 127.0.98.1:52745)
0620 09:03:54.288558 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.288599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.289743 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.290294 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.290300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.290643 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:54.291097 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.291109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.291923 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.291927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.292935 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.292944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.293481 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.293484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.293639 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.294301 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:54.294317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:54.295713 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:54.297692 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.297698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.297712 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.297961 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.298250 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.298254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.298256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.298404 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.298800 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:54.298807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:54.299051 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.299233 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.299539 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:03:54.301594 18711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36771
I20260620 09:03:54.301798 18711 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:54.302268 18711 heartbeater.cc:511] Master 127.0.98.62:36771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:03:54.303824 18520 ts_manager.cc:194] Registered new tserver with Master: ff07728b273e4f8a8c0f67ca19c21791 (127.0.98.1:34189)
I20260620 09:03:54.305372 18520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:03:54.319131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:54.426187 18716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:54.426481 18716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:54.426554 18716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:54.430595 18716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:54.430673 18716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:54.430759 18716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:54.435846 18716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:54.436893 18716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:54.437872 18716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:54.446920 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:54.446939 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:54.447005 18716 server_base.cc:1034] running on GCE node
W20260620 09:03:54.446920 18724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:54.447531 18716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:54.448110 18716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:54.449265 18716 hybrid_clock.cc:648] HybridClock initialized: now 1781946234449250 us; error 26 us; skew 500 ppm
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:54.455753 18716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:03:54.456691 18716 webserver.cc:460] Webserver started at http://127.0.98.2:38741/ using document root <none> and password file <none>
I20260620 09:03:54.457252 18716 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:54.457311 18716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:54.457495 18716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:54.459514 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "c903e1c30d29425b9463d28e165fe89e"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "ae8798ea1162e17824f15787d2635816"
server_key_iv: "b81825959fbf470bd9d64e5b0d12a16b"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.460081 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c903e1c30d29425b9463d28e165fe89e"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "ae8798ea1162e17824f15787d2635816"
server_key_iv: "b81825959fbf470bd9d64e5b0d12a16b"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.463678 18716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:03:54.466202 18731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.467420 18716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:54.467561 18716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "c903e1c30d29425b9463d28e165fe89e"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "ae8798ea1162e17824f15787d2635816"
server_key_iv: "b81825959fbf470bd9d64e5b0d12a16b"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.467655 18716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:54.480433 18716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:54.483419 18716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:54.483613 18716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:54.484184 18716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:54.484980 18716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:54.485035 18716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.485098 18716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:54.485131 18716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.493624 18716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38983
I20260620 09:03:54.493657 18844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38983 every 8 connection(s)
I20260620 09:03:54.494477 18716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:54.504694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18716
I20260620 09:03:54.504845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:54.505122 392 external_mini_cluster.cc:1428] Setting key 84adb2c03b48cb520edb7dadf849723c
I20260620 09:03:54.507648 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.496159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60967 (local address 127.0.98.62:36771)
0620 09:03:54.496325 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.496330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.497119 (+ 789us) server_negotiation.cc:407] Connection header received
0620 09:03:54.498308 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.498311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.498354 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.498427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:54.499514 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.500088 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.501292 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.501519 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.501642 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.503958 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:54.503961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:54.503963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:54.503985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:54.505443 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.506082 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.506085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.506087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.506167 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.506561 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.506564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.506570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.506786 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:54.506886 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.507343 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.507491 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 09:03:54.508858 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.496553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36771 (local address 127.0.98.2:60967)
0620 09:03:54.496975 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.497015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.498120 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.498588 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.498594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.498934 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:54.499385 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.499397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.500189 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.500192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.501159 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.501167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.501697 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.501700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.501835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.502412 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:54.502428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:54.503812 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:54.505551 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.505556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.505570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.505977 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.506256 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.506260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.506262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.506446 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.506885 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:54.506892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:54.507125 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.507304 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.507688 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260620 09:03:54.509941 18845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36771
I20260620 09:03:54.510186 18845 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:54.510757 18845 heartbeater.cc:511] Master 127.0.98.62:36771 requested a full tablet report, sending...
I20260620 09:03:54.511845 18520 ts_manager.cc:194] Registered new tserver with Master: c903e1c30d29425b9463d28e165fe89e (127.0.98.2:38983)
I20260620 09:03:54.512427 18520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60967
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:03:54.532217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:54.638943 18850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:54.639219 18850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:54.639292 18850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:54.643319 18850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:54.643378 18850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:54.643455 18850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:54.649477 18850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:54.650434 18850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:54.651391 18850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:54.660501 18856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:54.660542 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:54.660616 18850 server_base.cc:1034] running on GCE node
W20260620 09:03:54.660517 18858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:54.661113 18850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:54.661720 18850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:54.662899 18850 hybrid_clock.cc:648] HybridClock initialized: now 1781946234662886 us; error 27 us; skew 500 ppm
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:54.665550 18850 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:03:54.666364 18850 webserver.cc:460] Webserver started at http://127.0.98.3:32983/ using document root <none> and password file <none>
I20260620 09:03:54.666837 18850 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:54.666891 18850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:54.667094 18850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:54.669000 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "9873ecc6c4c74941940ef7f45bd3527a"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "41de638c95377d7d59fffbdd29cd37ce"
server_key_iv: "ebdde17acbefcc20528ff9b6d0327aa8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.669497 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9873ecc6c4c74941940ef7f45bd3527a"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "41de638c95377d7d59fffbdd29cd37ce"
server_key_iv: "ebdde17acbefcc20528ff9b6d0327aa8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.672876 18850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:54.674997 18865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.676069 18850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:54.676170 18850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "9873ecc6c4c74941940ef7f45bd3527a"
format_stamp: "Formatted at 2026-06-20 09:03:54 on dist-test-slave-scsf"
server_key: "41de638c95377d7d59fffbdd29cd37ce"
server_key_iv: "ebdde17acbefcc20528ff9b6d0327aa8"
server_key_version: "encryptionkey@0"
I20260620 09:03:54.676249 18850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:54.708381 18850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:54.711534 18850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:54.711683 18850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:54.712184 18850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:54.712965 18850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:54.713016 18850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.713054 18850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:54.713079 18850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:54.721596 18850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37061
I20260620 09:03:54.721623 18978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37061 every 8 connection(s)
I20260620 09:03:54.722535 18850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:54.728017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 18850
I20260620 09:03:54.728193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:54.728516 392 external_mini_cluster.cc:1428] Setting key 6bf449a6bf1d575773d5d1f703e71de4
Jun 20 09:03:54 dist-test-slave-scsf krb5kdc[18475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:54.736239 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.724297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56313 (local address 127.0.98.62:36771)
0620 09:03:54.724428 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.724432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.725530 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 09:03:54.726798 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.726801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.726848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.726924 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:54.728131 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.728833 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.730189 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.730479 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.730597 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.732957 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:54.732960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:54.732963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:54.732988 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:54.734289 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.734811 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.734814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.734816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.734900 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.735250 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.735254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.735256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.735431 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:54.735529 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.735972 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.736085 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:03:54.737104 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.724735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36771 (local address 127.0.98.3:56313)
0620 09:03:54.725189 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.725228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.726589 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.727066 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.727073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.727399 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:54.727935 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.727955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.728971 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.728974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.730073 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.730084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.730651 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.730655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.730821 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.731385 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:54.731400 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:54.732804 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:54.734439 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.734445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.734458 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.734703 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.734991 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:54.734993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:54.734995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:54.735151 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:54.735528 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:54.735534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:54.735754 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.735944 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.736250 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260620 09:03:54.738121 18979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36771
I20260620 09:03:54.738359 18979 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:54.738816 18979 heartbeater.cc:511] Master 127.0.98.62:36771 requested a full tablet report, sending...
I20260620 09:03:54.739737 18520 ts_manager.cc:194] Registered new tserver with Master: 9873ecc6c4c74941940ef7f45bd3527a (127.0.98.3:37061)
I20260620 09:03:54.740284 18520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56313
I20260620 09:03:54.742317 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:54.751458 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.744143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.98.62:36771)
0620 09:03:54.744282 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.744287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.744422 (+ 135us) server_negotiation.cc:407] Connection header received
0620 09:03:54.744565 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.744569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.744605 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.744669 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:54.745304 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.745873 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.746637 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.746847 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.746920 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.748262 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:54.748265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:54.748267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:54.748287 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:54.749733 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.750237 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.750245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.750247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.750315 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:54.750631 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:54.750636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:54.750637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:54.750827 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:54.750974 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.751226 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.751316 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:03:54.754153 18520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:54.757089 18520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:54.770885 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.765680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.0.98.1:34189)
0620 09:03:54.766043 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.766056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.766272 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:03:54.766460 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.766465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.766737 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.766843 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:54.767238 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.768311 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.769698 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.770217 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.770329 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.770473 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.770666 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.770762 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
I20260620 09:03:54.770898 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.765433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34189 (local address 127.0.0.1:41736)
0620 09:03:54.766180 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.766194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.766287 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.767015 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.767018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.767035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:54.767114 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.767120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.768421 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.768427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.769581 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.769590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.770328 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.770331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.770425 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.770514 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.770647 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.770758 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":48,"thread_start_us":89,"threads_started":1}
I20260620 09:03:54.771550 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.764817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38983 (local address 127.0.0.1:51574)
0620 09:03:54.766038 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.766052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.766142 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.766932 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.766935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.766952 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:54.767044 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.767051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.768001 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.768007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.769569 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.769579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.771039 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.771042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.771141 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.771163 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.771255 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.771421 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":94,"thread_start_us":101,"threads_started":1}
I20260620 09:03:54.771823 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.764775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37061 (local address 127.0.0.1:41570)
0620 09:03:54.765501 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.765546 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.765671 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.766420 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.766426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.766447 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:54.766597 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.766604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.768228 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.768231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.769569 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.769579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.771433 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.771436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.771527 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.771559 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.771613 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.771687 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":46,"thread_start_us":78,"threads_started":1}
I20260620 09:03:54.772183 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.765003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.0.98.2:38983)
0620 09:03:54.765334 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.765347 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.766400 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 09:03:54.766574 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.766579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.766790 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.766942 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:54.767165 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.767885 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.769707 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.770220 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.770317 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.770413 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.771946 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.772037 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
I20260620 09:03:54.772539 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.765510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.98.3:37061)
0620 09:03:54.765906 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.765918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.765945 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:03:54.766018 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.766024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.766255 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.766782 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:54.766939 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.768108 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.770530 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.771311 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.772081 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.772209 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.772254 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.772347 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":169,"threads_started":1}
I20260620 09:03:54.774102 18645 tablet_service.cc:1467] Processing CreateTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 (DEFAULT_TABLE table=test-table [id=9871095e137a45928a38dd5b8d6c438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:54.774400 18779 tablet_service.cc:1467] Processing CreateTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 (DEFAULT_TABLE table=test-table [id=9871095e137a45928a38dd5b8d6c438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:54.775154 18779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceb19f152de41c6ba4cc7862e1a2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:54.775180 18645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceb19f152de41c6ba4cc7862e1a2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:54.774973 18913 tablet_service.cc:1467] Processing CreateTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 (DEFAULT_TABLE table=test-table [id=9871095e137a45928a38dd5b8d6c438d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:54.775833 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceb19f152de41c6ba4cc7862e1a2dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:54.781648 18996 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Bootstrap starting.
I20260620 09:03:54.782531 18998 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: Bootstrap starting.
I20260620 09:03:54.784515 18998 tablet_bootstrap.cc:654] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:54.784802 18997 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Bootstrap starting.
I20260620 09:03:54.784833 18996 tablet_bootstrap.cc:654] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:54.785732 18998 log.cc:824] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:54.785732 18996 log.cc:824] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:54.786608 18997 tablet_bootstrap.cc:654] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:54.787684 18998 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: No bootstrap required, opened a new log
I20260620 09:03:54.787755 18997 log.cc:824] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:54.787901 18998 ts_tablet_manager.cc:1397] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:03:54.788210 18996 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: No bootstrap required, opened a new log
I20260620 09:03:54.788393 18996 ts_tablet_manager.cc:1397] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 09:03:54.789625 18997 tablet_bootstrap.cc:492] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: No bootstrap required, opened a new log
I20260620 09:03:54.789786 18997 ts_tablet_manager.cc:1397] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:03:54.790937 18998 raft_consensus.cc:348] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.791167 18998 raft_consensus.cc:374] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:54.791218 18998 raft_consensus.cc:729] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff07728b273e4f8a8c0f67ca19c21791, State: Initialized, Role: FOLLOWER
I20260620 09:03:54.791702 18998 consensus_queue.cc:260] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.792224 18996 raft_consensus.cc:348] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.792399 18711 heartbeater.cc:503] Master 127.0.98.62:36771 was elected leader, sending a full tablet report...
I20260620 09:03:54.792495 18996 raft_consensus.cc:374] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:54.792554 18996 raft_consensus.cc:729] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c903e1c30d29425b9463d28e165fe89e, State: Initialized, Role: FOLLOWER
I20260620 09:03:54.792706 18998 ts_tablet_manager.cc:1428] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:03:54.792986 18996 consensus_queue.cc:260] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.793867 18996 ts_tablet_manager.cc:1428] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260620 09:03:54.794180 18845 heartbeater.cc:503] Master 127.0.98.62:36771 was elected leader, sending a full tablet report...
I20260620 09:03:54.794446 18997 raft_consensus.cc:348] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.794629 18997 raft_consensus.cc:374] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:54.794659 18997 raft_consensus.cc:729] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9873ecc6c4c74941940ef7f45bd3527a, State: Initialized, Role: FOLLOWER
I20260620 09:03:54.795018 18997 consensus_queue.cc:260] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.795885 18979 heartbeater.cc:503] Master 127.0.98.62:36771 was elected leader, sending a full tablet report...
I20260620 09:03:54.795956 18997 ts_tablet_manager.cc:1428] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:03:54.959096 19002 raft_consensus.cc:482] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:54.959314 19002 raft_consensus.cc:504] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.960510 19002 leader_election.cc:288] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9873ecc6c4c74941940ef7f45bd3527a (127.0.98.3:37061), c903e1c30d29425b9463d28e165fe89e (127.0.98.2:38983)
I20260620 09:03:54.964668 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.960801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45381 (local address 127.0.98.3:37061)
0620 09:03:54.960939 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.960943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.961025 (+ 82us) server_negotiation.cc:407] Connection header received
0620 09:03:54.961220 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.961228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.961271 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.961352 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:54.961621 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.962127 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.963668 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.964056 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.964138 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.964208 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.964414 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.964526 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:03:54.964708 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.960984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49651 (local address 127.0.98.2:38983)
0620 09:03:54.961133 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:54.961138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:54.961252 (+ 114us) server_negotiation.cc:407] Connection header received
0620 09:03:54.961396 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:54.961399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:54.961445 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:54.961519 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:54.961788 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.962340 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.963718 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.964134 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.964237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.964300 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:54.964518 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 09:03:54.964592 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:03:54.964864 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.960683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37061 (local address 127.0.98.1:45381)
0620 09:03:54.960937 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.960952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.961108 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.961365 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.961368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.961385 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:54.961507 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.961511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.962229 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.962233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.963528 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.963538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.964186 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.964192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.964289 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.964314 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.964389 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.964483 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260620 09:03:54.964874 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:54.960757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38983 (local address 127.0.98.1:49651)
0620 09:03:54.961163 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:54.961180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:54.961295 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:54.961542 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:54.961545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:54.961560 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:54.961682 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.961687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.962441 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.962444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.963551 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:54.963557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:54.964254 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:54.964257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:54.964351 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:54.964392 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:03:54.964501 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:03:54.964565 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260620 09:03:54.965528 18799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceb19f152de41c6ba4cc7862e1a2dc8" candidate_uuid: "ff07728b273e4f8a8c0f67ca19c21791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c903e1c30d29425b9463d28e165fe89e" is_pre_election: true
I20260620 09:03:54.965528 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceb19f152de41c6ba4cc7862e1a2dc8" candidate_uuid: "ff07728b273e4f8a8c0f67ca19c21791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9873ecc6c4c74941940ef7f45bd3527a" is_pre_election: true
I20260620 09:03:54.965845 18933 raft_consensus.cc:2454] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff07728b273e4f8a8c0f67ca19c21791 in term 0.
I20260620 09:03:54.965845 18799 raft_consensus.cc:2454] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff07728b273e4f8a8c0f67ca19c21791 in term 0.
I20260620 09:03:54.966349 18599 leader_election.cc:302] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9873ecc6c4c74941940ef7f45bd3527a, c903e1c30d29425b9463d28e165fe89e, ff07728b273e4f8a8c0f67ca19c21791; no voters:
I20260620 09:03:54.966630 19002 raft_consensus.cc:2781] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:54.966725 19002 raft_consensus.cc:482] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:54.966784 19002 raft_consensus.cc:3037] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:54.967821 19002 raft_consensus.cc:504] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
I20260620 09:03:54.968147 19002 leader_election.cc:288] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [CANDIDATE]: Term 1 election: Requested vote from peers 9873ecc6c4c74941940ef7f45bd3527a (127.0.98.3:37061), c903e1c30d29425b9463d28e165fe89e (127.0.98.2:38983)
I20260620 09:03:54.968531 18799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceb19f152de41c6ba4cc7862e1a2dc8" candidate_uuid: "ff07728b273e4f8a8c0f67ca19c21791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c903e1c30d29425b9463d28e165fe89e"
I20260620 09:03:54.968535 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceb19f152de41c6ba4cc7862e1a2dc8" candidate_uuid: "ff07728b273e4f8a8c0f67ca19c21791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9873ecc6c4c74941940ef7f45bd3527a"
I20260620 09:03:54.968663 18799 raft_consensus.cc:3037] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:54.968675 18933 raft_consensus.cc:3037] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:54.969898 18799 raft_consensus.cc:2454] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff07728b273e4f8a8c0f67ca19c21791 in term 1.
I20260620 09:03:54.969899 18933 raft_consensus.cc:2454] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff07728b273e4f8a8c0f67ca19c21791 in term 1.
I20260620 09:03:54.970184 18599 leader_election.cc:302] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9873ecc6c4c74941940ef7f45bd3527a, c903e1c30d29425b9463d28e165fe89e, ff07728b273e4f8a8c0f67ca19c21791; no voters:
I20260620 09:03:54.970333 19002 raft_consensus.cc:2781] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:54.970556 19002 raft_consensus.cc:686] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [term 1 LEADER]: Becoming Leader. State: Replica: ff07728b273e4f8a8c0f67ca19c21791, State: Running, Role: LEADER
I20260620 09:03:54.970862 19002 consensus_queue.cc:237] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } }
W20260620 09:03:54.973430 18980 tablet.cc:2369] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:54.974095 18520 catalog_manager.cc:5495] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff07728b273e4f8a8c0f67ca19c21791 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ff07728b273e4f8a8c0f67ca19c21791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff07728b273e4f8a8c0f67ca19c21791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34189 } health_report { overall_health: HEALTHY } } }
W20260620 09:03:54.995708 18846 tablet.cc:2369] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:55.004755 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.001326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.0.98.1:34189)
0620 09:03:55.001464 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.001471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.001500 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:03:55.001571 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.001573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.001624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.001730 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:55.001972 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.002474 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.003287 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.003502 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.003605 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.003935 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:55.004297 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:55.004447 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.004588 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.004626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:03:55.010501 18799 raft_consensus.cc:1261] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 1 FOLLOWER]: Refusing update from remote peer ff07728b273e4f8a8c0f67ca19c21791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:55.010510 18933 raft_consensus.cc:1261] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 1 FOLLOWER]: Refusing update from remote peer ff07728b273e4f8a8c0f67ca19c21791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:55.011204 19007 consensus_queue.cc:1035] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9873ecc6c4c74941940ef7f45bd3527a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:55.011386 19002 consensus_queue.cc:1035] T cceb19f152de41c6ba4cc7862e1a2dc8 P ff07728b273e4f8a8c0f67ca19c21791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c903e1c30d29425b9463d28e165fe89e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:55.020258 19011 mvcc.cc:205] Tried to move back new op lower bound from 7298851778596163584 to 7298851778443337728. Current Snapshot: MvccSnapshot[applied={T|T < 7298851778596163584}]
I20260620 09:03:55.028296 18518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:55.028498 18518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:55.030941 18518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a648ce36af9b4809b1ac7d6da2c7ef96: Sending DeleteTablet for 3 replicas of tablet cceb19f152de41c6ba4cc7862e1a2dc8
I20260620 09:03:55.031694 18779 tablet_service.cc:1514] Processing DeleteTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51574
I20260620 09:03:55.031697 18913 tablet_service.cc:1514] Processing DeleteTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41570
I20260620 09:03:55.032151 19021 tablet_replica.cc:330] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: stopping tablet replica
I20260620 09:03:55.032385 19020 tablet_replica.cc:330] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: stopping tablet replica
I20260620 09:03:55.032514 19021 raft_consensus.cc:2229] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:55.032642 18645 tablet_service.cc:1514] Processing DeleteTablet for tablet cceb19f152de41c6ba4cc7862e1a2dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41736
I20260620 09:03:55.032828 19021 raft_consensus.cc:2258] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:55.032843 19020 raft_consensus.cc:2229] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:55.033062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18582
I20260620 09:03:55.033620 19020 raft_consensus.cc:2258] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:55.034597 19021 ts_tablet_manager.cc:1905] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:55.036269 19020 ts_tablet_manager.cc:1905] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:55.037449 19021 ts_tablet_manager.cc:1918] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:55.037549 19021 log.cc:1196] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/cceb19f152de41c6ba4cc7862e1a2dc8
I20260620 09:03:55.037894 19021 ts_tablet_manager.cc:1939] T cceb19f152de41c6ba4cc7862e1a2dc8 P c903e1c30d29425b9463d28e165fe89e: Deleting consensus metadata
I20260620 09:03:55.038825 19020 ts_tablet_manager.cc:1918] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:55.038919 19020 log.cc:1196] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/cceb19f152de41c6ba4cc7862e1a2dc8
I20260620 09:03:55.039119 18505 catalog_manager.cc:4841] TS c903e1c30d29425b9463d28e165fe89e (127.0.98.2:38983): tablet cceb19f152de41c6ba4cc7862e1a2dc8 (table test-table [id=9871095e137a45928a38dd5b8d6c438d]) successfully deleted
I20260620 09:03:55.039302 19020 ts_tablet_manager.cc:1939] T cceb19f152de41c6ba4cc7862e1a2dc8 P 9873ecc6c4c74941940ef7f45bd3527a: Deleting consensus metadata
W20260620 09:03:55.039918 18506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34189 (error 108)
I20260620 09:03:55.040134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18716
I20260620 09:03:55.040957 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.040623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34189 (local address 127.0.0.1:41750)
0620 09:03:55.040813 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.040878 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 09:03:55.041250 18506 catalog_manager.cc:4584] TS ff07728b273e4f8a8c0f67ca19c21791 (127.0.98.1:34189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cceb19f152de41c6ba4cc7862e1a2dc8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34189: connect: Connection refused (error 111)
I20260620 09:03:55.042130 18507 catalog_manager.cc:4841] TS 9873ecc6c4c74941940ef7f45bd3527a (127.0.98.3:37061): tablet cceb19f152de41c6ba4cc7862e1a2dc8 (table test-table [id=9871095e137a45928a38dd5b8d6c438d]) successfully deleted
I20260620 09:03:55.046007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18850
I20260620 09:03:55.051709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 18488
2026-06-20T09:03:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2406 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T09:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:55Z Disabled control of system clock
I20260620 09:03:55.081321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43947
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:55.188114 19027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:55.188338 19027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:55.188390 19027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:55.192248 19027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:55.192298 19027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:55.192313 19027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:55.192329 19027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:55.192343 19027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:55.196869 19027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:55.197751 19027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:55.198984 19027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:55.207041 19033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.207072 19032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:55.207055 19027 server_base.cc:1034] running on GCE node
W20260620 09:03:55.207093 19035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:55.207815 19027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:55.208869 19027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:55.210018 19027 hybrid_clock.cc:648] HybridClock initialized: now 1781946235210005 us; error 25 us; skew 500 ppm
I20260620 09:03:55.211795 19027 webserver.cc:460] Webserver started at http://127.0.98.62:39831/ using document root <none> and password file <none>
I20260620 09:03:55.212337 19027 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:55.212394 19027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:55.212687 19027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:55.214545 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "7013f17c9660e77a5c55579107c13515"
server_key_iv: "d506cf1e70bcb0aee6ed69ac0673b853"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.215114 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "7013f17c9660e77a5c55579107c13515"
server_key_iv: "d506cf1e70bcb0aee6ed69ac0673b853"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.218464 19027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 09:03:55.220707 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.221779 19027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:03:55.221911 19027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "7013f17c9660e77a5c55579107c13515"
server_key_iv: "d506cf1e70bcb0aee6ed69ac0673b853"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.222010 19027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:55.234318 19027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:55.234879 19027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:55.235051 19027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:55.241492 19027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43947
I20260620 09:03:55.241521 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43947 every 8 connection(s)
I20260620 09:03:55.242331 19027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:55.245216 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:55.246410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19027
I20260620 09:03:55.246551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:55.246827 392 external_mini_cluster.cc:1428] Setting key 5a39db56bc4acd50767f7dbb2deb1f3f
I20260620 09:03:55.251425 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: Bootstrap starting.
I20260620 09:03:55.253867 19094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:55.254590 19094 log.cc:824] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:55.256322 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: No bootstrap required, opened a new log
I20260620 09:03:55.256378 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.248134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.0.98.62:43947)
0620 09:03:55.248792 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.248810 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.248851 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:03:55.249697 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.249719 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.250191 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.250546 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:55.250951 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.252325 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.253199 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.253857 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.254017 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.254820 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:55.254824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:55.254836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:55.255194 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:55.255252 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:55.255270 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:55.255373 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.255494 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.255724 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260620 09:03:55.260217 19094 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } }
I20260620 09:03:55.260509 19094 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:55.260586 19094 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb6dc8137ec48bc92c5c0bdf66fa818, State: Initialized, Role: FOLLOWER
I20260620 09:03:55.260962 19094 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } }
I20260620 09:03:55.261096 19094 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:55.261157 19094 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:55.261251 19094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:55.262040 19094 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } }
I20260620 09:03:55.262356 19094 leader_election.cc:302] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb6dc8137ec48bc92c5c0bdf66fa818; no voters:
I20260620 09:03:55.262621 19094 leader_election.cc:288] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:55.262727 19099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:55.262943 19099 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [term 1 LEADER]: Becoming Leader. State: Replica: dcb6dc8137ec48bc92c5c0bdf66fa818, State: Running, Role: LEADER
I20260620 09:03:55.263221 19099 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } }
I20260620 09:03:55.263651 19094 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:55.264780 19101 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb6dc8137ec48bc92c5c0bdf66fa818. Latest consensus state: current_term: 1 leader_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } } }
I20260620 09:03:55.264780 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb6dc8137ec48bc92c5c0bdf66fa818" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43947 } } }
I20260620 09:03:55.264998 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:55.264963 19101 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:55.265404 19109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:55.267939 19109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:55.272940 19109 catalog_manager.cc:1347] Generated new cluster ID: 4fa33b5759f74556a63def708d179c2a
I20260620 09:03:55.273006 19109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:55.287842 19109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:55.288900 19109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:55.294981 19109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: Generated new TSK 0
I20260620 09:03:55.295557 19109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:03:55.300230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--builtin_ntp_servers=127.0.98.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:55.414026 19118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:55.414299 19118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:55.414337 19118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:55.418283 19118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:55.418329 19118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:55.418407 19118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:55.423082 19118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:55.424052 19118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:55.425467 19118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:55.434451 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.434459 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.434459 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:55.434836 19118 server_base.cc:1034] running on GCE node
I20260620 09:03:55.435222 19118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:55.435822 19118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:55.437021 19118 hybrid_clock.cc:648] HybridClock initialized: now 1781946235436998 us; error 31 us; skew 500 ppm
I20260620 09:03:55.438947 19118 webserver.cc:460] Webserver started at http://127.0.98.1:39645/ using document root <none> and password file <none>
I20260620 09:03:55.439523 19118 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:55.439618 19118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:55.439842 19118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:55.441846 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "b048b1248a1243e98274fbb016b4b911"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "bd941f207225dead78b4eda8a2fa6985"
server_key_iv: "17ea072caf84d9ffd558e4a155ee5faa"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.442385 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b048b1248a1243e98274fbb016b4b911"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "bd941f207225dead78b4eda8a2fa6985"
server_key_iv: "17ea072caf84d9ffd558e4a155ee5faa"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.445974 19118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 09:03:55.448215 19132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.449352 19118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:03:55.449469 19118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "b048b1248a1243e98274fbb016b4b911"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "bd941f207225dead78b4eda8a2fa6985"
server_key_iv: "17ea072caf84d9ffd558e4a155ee5faa"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.449599 19118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:55.478466 19118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:55.479235 19118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:55.479434 19118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:55.480050 19118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:55.480994 19118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:55.481060 19118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.481122 19118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:55.481161 19118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.489919 19118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41853
I20260620 09:03:55.489943 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41853 every 8 connection(s)
I20260620 09:03:55.490804 19118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:55.496963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19118
I20260620 09:03:55.497107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:55.497406 392 external_mini_cluster.cc:1428] Setting key 97be350a580ff487529ec78288d043af
I20260620 09:03:55.499238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--builtin_ntp_servers=127.0.98.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:55.500097 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.492550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43361 (local address 127.0.98.62:43947)
0620 09:03:55.492687 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.492691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.493749 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 09:03:55.495140 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.495143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.495201 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.495296 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:55.496156 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.496720 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.498066 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.498294 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.498399 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.499616 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:55.499619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:55.499621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:55.499632 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:55.499650 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:55.499657 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:55.499737 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.499899 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.499961 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:03:55.500962 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.492978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43947 (local address 127.0.98.1:43361)
0620 09:03:55.493604 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.493644 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.494916 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.495468 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.495475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.495558 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:55.496012 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.496023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.496846 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.496850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.497950 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.497966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.498434 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.498438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.498655 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.499311 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:55.499313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:55.499375 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:55.499454 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:55.499749 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:55.499759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:55.499764 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.499885 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.500145 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260620 09:03:55.502151 19246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43947
I20260620 09:03:55.502379 19246 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:55.502996 19246 heartbeater.cc:511] Master 127.0.98.62:43947 requested a full tablet report, sending...
I20260620 09:03:55.504771 19058 ts_manager.cc:194] Registered new tserver with Master: b048b1248a1243e98274fbb016b4b911 (127.0.98.1:41853)
I20260620 09:03:55.506572 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43361
W20260620 09:03:55.608237 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:55.608526 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:55.608577 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:55.612557 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:55.612603 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:55.612654 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:55.617254 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:55.618185 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:55.619438 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:55.628499 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.628575 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.628615 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:55.628755 19249 server_base.cc:1034] running on GCE node
I20260620 09:03:55.629145 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:55.629689 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:55.630882 19249 hybrid_clock.cc:648] HybridClock initialized: now 1781946235630864 us; error 31 us; skew 500 ppm
I20260620 09:03:55.632577 19249 webserver.cc:460] Webserver started at http://127.0.98.2:34827/ using document root <none> and password file <none>
I20260620 09:03:55.633126 19249 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:55.633211 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:55.633420 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:55.635331 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "6ba5a174c9ab41f2bec7639033a47ee3"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "25c644a55162792d450f929108cf400f"
server_key_iv: "eac9c66df52d3354c36687cc7ca32ef8"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.635871 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6ba5a174c9ab41f2bec7639033a47ee3"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "25c644a55162792d450f929108cf400f"
server_key_iv: "eac9c66df52d3354c36687cc7ca32ef8"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.639392 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 09:03:55.641662 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.642760 19249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:55.642880 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "6ba5a174c9ab41f2bec7639033a47ee3"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "25c644a55162792d450f929108cf400f"
server_key_iv: "eac9c66df52d3354c36687cc7ca32ef8"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.642982 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:55.663132 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:55.663834 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:55.664026 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:55.664633 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:55.665601 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:55.665665 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.665730 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:55.665776 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.674181 19249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34197
I20260620 09:03:55.674206 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34197 every 8 connection(s)
I20260620 09:03:55.675034 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:55.683791 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.676775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54453 (local address 127.0.98.62:43947)
0620 09:03:55.676906 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.676915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.677880 (+ 965us) server_negotiation.cc:407] Connection header received
0620 09:03:55.679252 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.679255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.679313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.679389 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:55.680225 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.680852 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.681965 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.682185 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.682292 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.683297 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:55.683300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:55.683303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:55.683316 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:55.683340 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:55.683348 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:55.683419 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.683575 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.683648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:03:55.684696 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.677138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43947 (local address 127.0.98.2:54453)
0620 09:03:55.677741 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.677780 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.679038 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.679544 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.679550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.679631 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:55.680092 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.680103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.680971 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.680975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.681838 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.681846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.682329 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.682333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.682491 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.683040 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:55.683041 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:55.683078 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:55.683149 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:55.683431 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:55.683441 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:55.683447 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.683573 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.683832 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260620 09:03:55.685233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19249
I20260620 09:03:55.685385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:55.685710 392 external_mini_cluster.cc:1428] Setting key 0fec6e8f7b4853076f25b8bb22e56a25
I20260620 09:03:55.685945 19377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43947
I20260620 09:03:55.686195 19377 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:55.686690 19377 heartbeater.cc:511] Master 127.0.98.62:43947 requested a full tablet report, sending...
I20260620 09:03:55.687603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--builtin_ntp_servers=127.0.98.20:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:55.687681 19058 ts_manager.cc:194] Registered new tserver with Master: 6ba5a174c9ab41f2bec7639033a47ee3 (127.0.98.2:34197)
I20260620 09:03:55.688246 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54453
W20260620 09:03:55.795257 19380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:55.795521 19380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:55.795590 19380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:55.799659 19380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:55.799731 19380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:55.799813 19380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:55.804406 19380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:55.805420 19380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:55.806725 19380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:55.815671 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.815717 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:55.815804 19388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:55.815709 19380 server_base.cc:1034] running on GCE node
I20260620 09:03:55.816223 19380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:55.816916 19380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:55.818094 19380 hybrid_clock.cc:648] HybridClock initialized: now 1781946235818079 us; error 25 us; skew 500 ppm
I20260620 09:03:55.819823 19380 webserver.cc:460] Webserver started at http://127.0.98.3:43923/ using document root <none> and password file <none>
I20260620 09:03:55.820343 19380 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:55.820399 19380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:55.820657 19380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:55.822695 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2d4b0dd3b32479a82598dc0fbdb19af"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "6b9991225c042e6d6a82483dceadaf9a"
server_key_iv: "bd42b107484e34796972490f12e89e0b"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.823204 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d4b0dd3b32479a82598dc0fbdb19af"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "6b9991225c042e6d6a82483dceadaf9a"
server_key_iv: "bd42b107484e34796972490f12e89e0b"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.826804 19380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:55.829056 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.830143 19380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:55.830246 19380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "b2d4b0dd3b32479a82598dc0fbdb19af"
format_stamp: "Formatted at 2026-06-20 09:03:55 on dist-test-slave-scsf"
server_key: "6b9991225c042e6d6a82483dceadaf9a"
server_key_iv: "bd42b107484e34796972490f12e89e0b"
server_key_version: "encryptionkey@0"
I20260620 09:03:55.830332 19380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:55.846378 19380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:55.847048 19380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:55.847230 19380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:55.847783 19380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:55.848754 19380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:55.848809 19380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.848881 19380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:55.848919 19380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:55.857522 19380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40241
I20260620 09:03:55.857551 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40241 every 8 connection(s)
I20260620 09:03:55.858385 19380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:55.863025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19380
I20260620 09:03:55.863163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:55.863432 392 external_mini_cluster.cc:1428] Setting key 41b3bb08762e044740a86217e48785b0
I20260620 09:03:55.867801 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.860061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40857 (local address 127.0.98.62:43947)
0620 09:03:55.860187 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.860191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.861227 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 09:03:55.862607 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.862610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.862668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.862763 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:55.864072 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.864869 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.866175 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.866383 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.866466 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.867408 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:55.867411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:55.867413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:55.867424 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:55.867443 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:55.867450 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:55.867506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.867647 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.867695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:03:55.868621 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.860439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43947 (local address 127.0.98.3:40857)
0620 09:03:55.861084 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.861131 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.862396 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.862930 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.862936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.863020 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:55.863603 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.863620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.865007 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.865011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.866069 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.866079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.866504 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.866507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.866656 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.867184 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:55.867185 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:55.867221 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:55.867279 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:55.867516 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:55.867522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:55.867526 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.867631 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.867834 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260620 09:03:55.869661 19508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43947
I20260620 09:03:55.869876 19508 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:55.870344 19508 heartbeater.cc:511] Master 127.0.98.62:43947 requested a full tablet report, sending...
I20260620 09:03:55.871282 19058 ts_manager.cc:194] Registered new tserver with Master: b2d4b0dd3b32479a82598dc0fbdb19af (127.0.98.3:40241)
I20260620 09:03:55.871778 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40857
I20260620 09:03:55.877039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:55.882202 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.878879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.0.98.62:43947)
0620 09:03:55.879092 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.879096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.879133 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:03:55.879230 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.879232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.879265 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.879328 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:55.879552 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.880043 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.880888 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.881088 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.881168 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.881856 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:55.881859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:55.881861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:55.881871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:55.881888 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:55.881895 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:55.881988 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.882073 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.882102 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260620 09:03:55.884794 19058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:55.887634 19058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:55.901533 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.895597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40241 (local address 127.0.0.1:33560)
0620 09:03:55.896388 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.896401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.896498 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.897457 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.897460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.897479 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:55.897639 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.897646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.898724 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.898727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.900045 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.900055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.901094 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.901181 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901206 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.901261 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.901385 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":82,"thread_start_us":85,"threads_started":1}
I20260620 09:03:55.901540 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.896299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.0.98.1:41853)
0620 09:03:55.896955 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.896967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.896991 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:03:55.897064 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.897074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.897222 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.897397 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:55.897753 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.898927 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.900284 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.900888 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901013 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901213 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.901281 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.901373 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":150,"threads_started":1}
I20260620 09:03:55.901845 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.896694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.98.3:40241)
0620 09:03:55.897079 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.897090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.897107 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:03:55.897167 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.897171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.897320 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.897791 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:55.897891 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.898610 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.900182 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.900952 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901365 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901517 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.901567 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.901676 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":95,"threads_started":1}
I20260620 09:03:55.902057 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.895822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41853 (local address 127.0.0.1:49622)
0620 09:03:55.896590 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.896603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.896671 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.897519 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.897541 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.897553 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:55.897657 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.897663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.899081 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.899085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.900156 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.900169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.901009 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.901114 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901149 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.901245 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.901854 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":60,"thread_start_us":91,"threads_started":1}
I20260620 09:03:55.902426 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.895612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.0.98.2:34197)
0620 09:03:55.896873 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:55.896885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:55.896908 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:03:55.896984 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:55.896989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:55.897200 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:55.897353 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:55.897678 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.898927 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.900501 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.901254 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901517 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:55.902172 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 09:03:55.902261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":110,"threads_started":1}
I20260620 09:03:55.903123 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:55.895489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34197 (local address 127.0.0.1:55914)
0620 09:03:55.896091 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:55.896153 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:55.896276 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:55.897357 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:55.897363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:55.897385 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:55.897551 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.897559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.899039 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.899043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.900376 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:55.900385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:55.901577 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:55.901581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:55.901873 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:55.901902 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:03:55.901974 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:03:55.902039 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":182,"thread_start_us":74,"threads_started":1}
I20260620 09:03:55.904830 19180 tablet_service.cc:1467] Processing CreateTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 (DEFAULT_TABLE table=test-table [id=2f38fd0367474b898ebfc1dba8f6bd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:55.904834 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 (DEFAULT_TABLE table=test-table [id=2f38fd0367474b898ebfc1dba8f6bd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:55.905750 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c69256f2d5946c791702bf4919e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:55.905812 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c69256f2d5946c791702bf4919e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:55.905529 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 (DEFAULT_TABLE table=test-table [id=2f38fd0367474b898ebfc1dba8f6bd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:55.906374 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c69256f2d5946c791702bf4919e0fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:55.911516 19525 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Bootstrap starting.
I20260620 09:03:55.913157 19526 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Bootstrap starting.
I20260620 09:03:55.914124 19525 tablet_bootstrap.cc:654] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:55.914808 19525 log.cc:824] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:55.914969 19526 tablet_bootstrap.cc:654] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:55.915416 19527 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Bootstrap starting.
I20260620 09:03:55.915735 19526 log.cc:824] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:55.916997 19527 tablet_bootstrap.cc:654] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:55.917431 19526 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: No bootstrap required, opened a new log
I20260620 09:03:55.917622 19526 ts_tablet_manager.cc:1397] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:03:55.917636 19527 log.cc:824] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:55.919080 19525 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: No bootstrap required, opened a new log
I20260620 09:03:55.919272 19525 ts_tablet_manager.cc:1397] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 09:03:55.919615 19527 tablet_bootstrap.cc:492] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: No bootstrap required, opened a new log
I20260620 09:03:55.919829 19527 ts_tablet_manager.cc:1397] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:03:55.921316 19526 raft_consensus.cc:348] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.921545 19526 raft_consensus.cc:374] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:55.921597 19526 raft_consensus.cc:729] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba5a174c9ab41f2bec7639033a47ee3, State: Initialized, Role: FOLLOWER
I20260620 09:03:55.922005 19526 consensus_queue.cc:260] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.922681 19377 heartbeater.cc:503] Master 127.0.98.62:43947 was elected leader, sending a full tablet report...
I20260620 09:03:55.922928 19526 ts_tablet_manager.cc:1428] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:03:55.923400 19525 raft_consensus.cc:348] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.923636 19525 raft_consensus.cc:374] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:55.923687 19525 raft_consensus.cc:729] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b048b1248a1243e98274fbb016b4b911, State: Initialized, Role: FOLLOWER
I20260620 09:03:55.924124 19525 consensus_queue.cc:260] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.924760 19246 heartbeater.cc:503] Master 127.0.98.62:43947 was elected leader, sending a full tablet report...
I20260620 09:03:55.925027 19525 ts_tablet_manager.cc:1428] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:03:55.924973 19527 raft_consensus.cc:348] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.925189 19527 raft_consensus.cc:374] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:55.925241 19527 raft_consensus.cc:729] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d4b0dd3b32479a82598dc0fbdb19af, State: Initialized, Role: FOLLOWER
W20260620 09:03:55.925890 19378 tablet.cc:2369] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:55.925709 19527 consensus_queue.cc:260] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:55.927379 19527 ts_tablet_manager.cc:1428] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 09:03:55.927712 19508 heartbeater.cc:503] Master 127.0.98.62:43947 was elected leader, sending a full tablet report...
W20260620 09:03:55.991943 19247 tablet.cc:2369] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:03:56.109313 19509 tablet.cc:2369] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:56.219002 19532 raft_consensus.cc:482] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:56.219362 19532 raft_consensus.cc:504] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:56.220578 19532 leader_election.cc:288] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ba5a174c9ab41f2bec7639033a47ee3 (127.0.98.2:34197), b2d4b0dd3b32479a82598dc0fbdb19af (127.0.98.3:40241)
I20260620 09:03:56.224398 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.220954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49467 (local address 127.0.98.2:34197)
0620 09:03:56.221089 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:56.221096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:56.221256 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:03:56.221402 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:56.221407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:56.221465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:56.221529 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:56.221825 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.222339 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.223461 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.223849 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.223958 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:56.224024 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:56.224194 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 09:03:56.224270 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:03:56.224401 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.220838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34197 (local address 127.0.98.1:49467)
0620 09:03:56.221151 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:56.221166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:56.221274 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:56.221561 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:56.221565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:56.221585 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:56.221723 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:56.221729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.222443 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.222447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:56.223342 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:56.223351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.223965 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.223969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:56.224065 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:56.224097 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 09:03:56.224157 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:03:56.224235 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":42,"thread_start_us":74,"threads_started":1}
I20260620 09:03:56.224413 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.220962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60491 (local address 127.0.98.3:40241)
0620 09:03:56.221151 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:56.221157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:56.221319 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:03:56.221460 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:56.221462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:56.221500 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:56.221589 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:56.221875 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.222341 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.223460 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.223862 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.223975 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:56.224033 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:56.224244 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 09:03:56.224312 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 09:03:56.224418 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.220838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40241 (local address 127.0.98.1:60491)
0620 09:03:56.221199 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:56.221209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:56.221360 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:56.221626 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:56.221630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:56.221653 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:56.221753 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:56.221757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.222447 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.222450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:56.223341 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:56.223350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.224010 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.224013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:56.224110 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:56.224127 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:03:56.224220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:03:56.224281 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":87,"thread_start_us":65,"threads_started":1}
I20260620 09:03:56.225153 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c69256f2d5946c791702bf4919e0fa7" candidate_uuid: "b048b1248a1243e98274fbb016b4b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" is_pre_election: true
I20260620 09:03:56.225481 19462 raft_consensus.cc:2454] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b048b1248a1243e98274fbb016b4b911 in term 0.
I20260620 09:03:56.225441 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c69256f2d5946c791702bf4919e0fa7" candidate_uuid: "b048b1248a1243e98274fbb016b4b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" is_pre_election: true
I20260620 09:03:56.225751 19331 raft_consensus.cc:2454] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b048b1248a1243e98274fbb016b4b911 in term 0.
I20260620 09:03:56.225986 19133 leader_election.cc:302] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b048b1248a1243e98274fbb016b4b911, b2d4b0dd3b32479a82598dc0fbdb19af; no voters:
I20260620 09:03:56.226276 19532 raft_consensus.cc:2781] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:56.226349 19532 raft_consensus.cc:482] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:56.226382 19532 raft_consensus.cc:3037] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:56.227351 19532 raft_consensus.cc:504] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:56.227699 19532 leader_election.cc:288] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [CANDIDATE]: Term 1 election: Requested vote from peers 6ba5a174c9ab41f2bec7639033a47ee3 (127.0.98.2:34197), b2d4b0dd3b32479a82598dc0fbdb19af (127.0.98.3:40241)
I20260620 09:03:56.227998 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c69256f2d5946c791702bf4919e0fa7" candidate_uuid: "b048b1248a1243e98274fbb016b4b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba5a174c9ab41f2bec7639033a47ee3"
I20260620 09:03:56.228019 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c69256f2d5946c791702bf4919e0fa7" candidate_uuid: "b048b1248a1243e98274fbb016b4b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af"
I20260620 09:03:56.228125 19331 raft_consensus.cc:3037] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:56.228130 19462 raft_consensus.cc:3037] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:56.229151 19331 raft_consensus.cc:2454] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b048b1248a1243e98274fbb016b4b911 in term 1.
I20260620 09:03:56.229235 19462 raft_consensus.cc:2454] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b048b1248a1243e98274fbb016b4b911 in term 1.
I20260620 09:03:56.229458 19134 leader_election.cc:302] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ba5a174c9ab41f2bec7639033a47ee3, b048b1248a1243e98274fbb016b4b911; no voters:
I20260620 09:03:56.229619 19532 raft_consensus.cc:2781] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:56.229854 19532 raft_consensus.cc:686] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [term 1 LEADER]: Becoming Leader. State: Replica: b048b1248a1243e98274fbb016b4b911, State: Running, Role: LEADER
I20260620 09:03:56.230173 19532 consensus_queue.cc:237] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } }
I20260620 09:03:56.233202 19056 catalog_manager.cc:5495] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 reported cstate change: term changed from 0 to 1, leader changed from <none> to b048b1248a1243e98274fbb016b4b911 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b048b1248a1243e98274fbb016b4b911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b048b1248a1243e98274fbb016b4b911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41853 } health_report { overall_health: HEALTHY } } }
I20260620 09:03:56.265407 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.261715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.0.98.1:41853)
0620 09:03:56.261870 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:56.261874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:56.261889 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:03:56.262025 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:56.262028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:56.262082 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:56.262203 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:56.262434 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.262930 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.263810 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:56.264066 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:56.264177 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:56.264528 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:56.264996 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:56.265142 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:56.265232 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 09:03:56.265267 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:03:56.271023 19331 raft_consensus.cc:1261] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 1 FOLLOWER]: Refusing update from remote peer b048b1248a1243e98274fbb016b4b911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:56.271023 19462 raft_consensus.cc:1261] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 1 FOLLOWER]: Refusing update from remote peer b048b1248a1243e98274fbb016b4b911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:56.271739 19532 consensus_queue.cc:1035] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba5a174c9ab41f2bec7639033a47ee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:56.271960 19536 consensus_queue.cc:1035] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d4b0dd3b32479a82598dc0fbdb19af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:56.289664 19057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:56.289832 19057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:56.292089 19057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcb6dc8137ec48bc92c5c0bdf66fa818: Sending DeleteTablet for 3 replicas of tablet 8c69256f2d5946c791702bf4919e0fa7
I20260620 09:03:56.292843 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:56 UTC) from {username='slave'} at 127.0.0.1:33560
I20260620 09:03:56.292927 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:56 UTC) from {username='slave'} at 127.0.0.1:55914
I20260620 09:03:56.292954 19180 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c69256f2d5946c791702bf4919e0fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:56 UTC) from {username='slave'} at 127.0.0.1:49622
I20260620 09:03:56.293462 19551 tablet_replica.cc:330] T 8c69256f2d5946c791702bf4919e0fa7 P b048b1248a1243e98274fbb016b4b911: stopping tablet replica
I20260620 09:03:56.293563 19549 tablet_replica.cc:330] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: stopping tablet replica
I20260620 09:03:56.293839 19549 raft_consensus.cc:2229] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:56.294059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19118
I20260620 09:03:56.295116 19550 tablet_replica.cc:330] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: stopping tablet replica
I20260620 09:03:56.295320 19550 raft_consensus.cc:2229] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:56.295533 19550 raft_consensus.cc:2258] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:56.295934 19549 raft_consensus.cc:2258] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:56.296974 19550 ts_tablet_manager.cc:1905] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:56.297150 19549 ts_tablet_manager.cc:1905] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:56.299199 19549 ts_tablet_manager.cc:1918] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:56.299309 19549 log.cc:1196] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/8c69256f2d5946c791702bf4919e0fa7
I20260620 09:03:56.299574 19549 ts_tablet_manager.cc:1939] T 8c69256f2d5946c791702bf4919e0fa7 P 6ba5a174c9ab41f2bec7639033a47ee3: Deleting consensus metadata
I20260620 09:03:56.300413 19550 ts_tablet_manager.cc:1918] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:56.300534 19550 log.cc:1196] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/8c69256f2d5946c791702bf4919e0fa7
I20260620 09:03:56.300618 19043 catalog_manager.cc:4841] TS 6ba5a174c9ab41f2bec7639033a47ee3 (127.0.98.2:34197): tablet 8c69256f2d5946c791702bf4919e0fa7 (table test-table [id=2f38fd0367474b898ebfc1dba8f6bd79]) successfully deleted
I20260620 09:03:56.300841 19550 ts_tablet_manager.cc:1939] T 8c69256f2d5946c791702bf4919e0fa7 P b2d4b0dd3b32479a82598dc0fbdb19af: Deleting consensus metadata
W20260620 09:03:56.301496 19044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41853 (error 108)
I20260620 09:03:56.301963 19042 catalog_manager.cc:4841] TS b2d4b0dd3b32479a82598dc0fbdb19af (127.0.98.3:40241): tablet 8c69256f2d5946c791702bf4919e0fa7 (table test-table [id=2f38fd0367474b898ebfc1dba8f6bd79]) successfully deleted
I20260620 09:03:56.302299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19249
I20260620 09:03:56.302465 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:56.302182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41853 (local address 127.0.0.1:49644)
0620 09:03:56.302301 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:56.302364 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 09:03:56.303668 19044 catalog_manager.cc:4584] TS b048b1248a1243e98274fbb016b4b911 (127.0.98.1:41853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c69256f2d5946c791702bf4919e0fa7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41853: connect: Connection refused (error 111)
I20260620 09:03:56.307760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19380
I20260620 09:03:56.313062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19027
2026-06-20T09:03:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1260 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:03:56 dist-test-slave-scsf krb5kdc[19555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:03:56 dist-test-slave-scsf krb5kdc[19555](info): set up 2 sockets
Jun 20 09:03:56 dist-test-slave-scsf krb5kdc[19555](info): commencing operation
krb5kdc: starting...
W20260620 09:03:57.348321 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:03:57 dist-test-slave-scsf krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:03:57.465695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:57.572350 19568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:57.572682 19568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:57.572762 19568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:57.576587 19568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:57.576650 19568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:57.576685 19568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:57.576718 19568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:57.576752 19568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:57.581841 19568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:57.582823 19568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:57.583784 19568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:57.591969 19576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:57.591979 19574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:57.591979 19573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:57.592149 19568 server_base.cc:1034] running on GCE node
I20260620 09:03:57.592828 19568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:57.593827 19568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:57.594976 19568 hybrid_clock.cc:648] HybridClock initialized: now 1781946237594963 us; error 24 us; skew 500 ppm
Jun 20 09:03:57 dist-test-slave-scsf krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:57.597569 19568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:03:57.598361 19568 webserver.cc:460] Webserver started at http://127.0.98.62:32913/ using document root <none> and password file <none>
I20260620 09:03:57.598891 19568 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:57.598950 19568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:57.599193 19568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:57.601192 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "c98fb68f50bb4f52bb293d7a828076f8"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "b70ce33de382d68ef1f3b4d4b757bfb0"
server_key_iv: "1319e77d7626f6dbe3f0342799270df6"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.601725 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "c98fb68f50bb4f52bb293d7a828076f8"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "b70ce33de382d68ef1f3b4d4b757bfb0"
server_key_iv: "1319e77d7626f6dbe3f0342799270df6"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.605269 19568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:57.607555 19583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:57.608824 19568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:57.608961 19568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "c98fb68f50bb4f52bb293d7a828076f8"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "b70ce33de382d68ef1f3b4d4b757bfb0"
server_key_iv: "1319e77d7626f6dbe3f0342799270df6"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.609062 19568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:57.623735 19568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:57.626375 19568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:57.626583 19568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:57.633379 19568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43977
I20260620 09:03:57.633399 19635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43977 every 8 connection(s)
I20260620 09:03:57.634323 19568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:57.636905 19636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:57.641093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19568
I20260620 09:03:57.641252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:57.641520 392 external_mini_cluster.cc:1428] Setting key 9d26c917c9a8fca4dbd99efe9d7d959a
I20260620 09:03:57.642268 19636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: Bootstrap starting.
I20260620 09:03:57.644496 19636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:57.645224 19636 log.cc:824] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:57.646983 19636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: No bootstrap required, opened a new log
Jun 20 09:03:57 dist-test-slave-scsf krb5kdc[19555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:57.650802 19636 raft_consensus.cc:348] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260620 09:03:57.651106 19636 raft_consensus.cc:374] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:57.651166 19636 raft_consensus.cc:729] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98fb68f50bb4f52bb293d7a828076f8, State: Initialized, Role: FOLLOWER
I20260620 09:03:57.651672 19636 consensus_queue.cc:260] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260620 09:03:57.651850 19636 raft_consensus.cc:388] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:57.651918 19636 raft_consensus.cc:482] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:57.652009 19636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:57.653117 19636 raft_consensus.cc:504] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260620 09:03:57.653471 19636 leader_election.cc:302] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c98fb68f50bb4f52bb293d7a828076f8; no voters:
I20260620 09:03:57.653800 19636 leader_election.cc:288] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:57.653910 19641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:57.654091 19641 raft_consensus.cc:686] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [term 1 LEADER]: Becoming Leader. State: Replica: c98fb68f50bb4f52bb293d7a828076f8, State: Running, Role: LEADER
I20260620 09:03:57.654405 19641 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } }
I20260620 09:03:57.655040 19636 sys_catalog.cc:564] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:57.655581 19641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c98fb68f50bb4f52bb293d7a828076f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } } }
I20260620 09:03:57.655710 19641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:57.655929 19641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c98fb68f50bb4f52bb293d7a828076f8. Latest consensus state: current_term: 1 leader_uuid: "c98fb68f50bb4f52bb293d7a828076f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98fb68f50bb4f52bb293d7a828076f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43977 } } }
I20260620 09:03:57.656052 19641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:57.656117 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:57.642826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.0.98.62:43977)
0620 09:03:57.643218 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:57.643232 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:57.643264 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:03:57.643846 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:57.643868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:57.644179 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:57.644517 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:57.645315 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.646318 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.647168 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.647844 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.648013 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:57.650406 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:57.650412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:57.650427 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:57.650456 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:57.652502 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:57.652907 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:57.652912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:57.652919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:57.653067 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:57.653372 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:57.653375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:57.653377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:57.653828 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:57.654010 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:57.654817 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 09:03:57.655124 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260620 09:03:57.656488 19647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:57.659699 19647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:57.664824 19647 catalog_manager.cc:1347] Generated new cluster ID: 5224f5fe55a94168bc08728c7b422e66
I20260620 09:03:57.664885 19647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:57.673390 19647 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:57.674425 19647 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:57.687674 19647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: Generated new TSK 0
I20260620 09:03:57.688547 19647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:03:57.715778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:57.828089 19662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:57.828377 19662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:57.828490 19662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:57.832396 19662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:57.832499 19662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:57.832603 19662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:57.837798 19662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:57.838851 19662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:57.839870 19662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:57.849404 19667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:57.849412 19668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:57.849403 19670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:57.849692 19662 server_base.cc:1034] running on GCE node
I20260620 09:03:57.850159 19662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:57.850792 19662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:57.851965 19662 hybrid_clock.cc:648] HybridClock initialized: now 1781946237851931 us; error 49 us; skew 500 ppm
Jun 20 09:03:57 dist-test-slave-scsf krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:57.854593 19662 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:03:57.855480 19662 webserver.cc:460] Webserver started at http://127.0.98.1:41617/ using document root <none> and password file <none>
I20260620 09:03:57.856129 19662 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:57.856212 19662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:57.856426 19662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:57.858476 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "bada63af68f94c32bd342853af94f86c"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "dc85b20dedc92cd6f1d4dc6300746992"
server_key_iv: "27c53611df2d41b58365cc4f8b6d20fe"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.859030 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bada63af68f94c32bd342853af94f86c"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "dc85b20dedc92cd6f1d4dc6300746992"
server_key_iv: "27c53611df2d41b58365cc4f8b6d20fe"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.862582 19662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:57.864881 19677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:57.866079 19662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:57.866204 19662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "bada63af68f94c32bd342853af94f86c"
format_stamp: "Formatted at 2026-06-20 09:03:57 on dist-test-slave-scsf"
server_key: "dc85b20dedc92cd6f1d4dc6300746992"
server_key_iv: "27c53611df2d41b58365cc4f8b6d20fe"
server_key_version: "encryptionkey@0"
I20260620 09:03:57.866319 19662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:57.885710 19662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:57.888581 19662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:57.888775 19662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:57.889514 19662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:57.890336 19662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:57.890422 19662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:57.890488 19662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:57.890518 19662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:57.899372 19662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38283
I20260620 09:03:57.899403 19790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38283 every 8 connection(s)
I20260620 09:03:57.900255 19662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:57.902356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19662
I20260620 09:03:57.902505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:57.902799 392 external_mini_cluster.cc:1428] Setting key f6af9827c7e306fcdbfef6492a5e43b8
Jun 20 09:03:57 dist-test-slave-scsf krb5kdc[19555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:03:57.914278 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:57.902159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35197 (local address 127.0.98.62:43977)
0620 09:03:57.902334 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:57.902338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:57.903288 (+ 950us) server_negotiation.cc:407] Connection header received
0620 09:03:57.904243 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:57.904246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:57.904297 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:57.904385 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:57.905507 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.906406 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.907551 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.907978 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.908106 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:57.910473 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:57.910480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:57.910483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:57.910514 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:57.912171 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:57.912784 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:57.912787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:57.912788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:57.912892 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:57.913286 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:57.913289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:57.913290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:57.913525 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:57.913601 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:57.914051 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 09:03:57.914159 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:03:57.915138 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:57.902588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43977 (local address 127.0.98.1:35197)
0620 09:03:57.903122 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:57.903181 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:57.904049 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:57.904569 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:57.904576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:57.904917 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:57.905372 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:57.905384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.906539 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.906541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:57.907429 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:57.907438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:57.908128 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:57.908131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:57.908268 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:57.908911 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:57.908926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:57.910311 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:57.912350 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:57.912355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:57.912370 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:57.912688 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:57.912989 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:57.912991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:57.912992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:57.913194 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:57.913609 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:57.913613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:57.913854 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 09:03:57.913997 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:03:57.914251 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:03:57.916368 19791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43977
I20260620 09:03:57.916648 19791 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:57.917177 19791 heartbeater.cc:511] Master 127.0.98.62:43977 requested a full tablet report, sending...
I20260620 09:03:57.918658 19600 ts_manager.cc:194] Registered new tserver with Master: bada63af68f94c32bd342853af94f86c (127.0.98.1:38283)
I20260620 09:03:57.919785 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:03:57.931006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:58.039986 19796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:58.040252 19796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:58.040297 19796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:58.044243 19796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:58.044301 19796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:58.044359 19796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:58.049584 19796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:58.050565 19796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:58.051563 19796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:58.060755 19802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:58.060766 19801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:58.060755 19804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:58.060838 19796 server_base.cc:1034] running on GCE node
I20260620 09:03:58.061364 19796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:58.061975 19796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:58.063149 19796 hybrid_clock.cc:648] HybridClock initialized: now 1781946238063120 us; error 48 us; skew 500 ppm
Jun 20 09:03:58 dist-test-slave-scsf krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:58.065778 19796 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:03:58.066601 19796 webserver.cc:460] Webserver started at http://127.0.98.2:37513/ using document root <none> and password file <none>
I20260620 09:03:58.067137 19796 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:58.067224 19796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:58.067437 19796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:58.069522 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "43cf0f8de34f47eaab9c87867cbc499c"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "87e40cc1eedacb1f3fd6b2050230d865"
server_key_iv: "0131908f28b7208e0a526941ed2cc565"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.070057 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43cf0f8de34f47eaab9c87867cbc499c"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "87e40cc1eedacb1f3fd6b2050230d865"
server_key_iv: "0131908f28b7208e0a526941ed2cc565"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.073679 19796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:03:58.076099 19811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.077418 19796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:58.077549 19796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "43cf0f8de34f47eaab9c87867cbc499c"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "87e40cc1eedacb1f3fd6b2050230d865"
server_key_iv: "0131908f28b7208e0a526941ed2cc565"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.077658 19796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:58.093108 19796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:58.096658 19796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:58.096875 19796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:58.097703 19796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:58.098556 19796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:58.098605 19796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.098645 19796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:58.098699 19796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.107466 19796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40281
I20260620 09:03:58.107513 19924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40281 every 8 connection(s)
I20260620 09:03:58.108376 19796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:03:58 dist-test-slave-scsf krb5kdc[19555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:58.117364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19796
I20260620 09:03:58.117493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:58.117836 392 external_mini_cluster.cc:1428] Setting key adce26ebc4f0e13515fc982f281af24f
I20260620 09:03:58.121690 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.110116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44919 (local address 127.0.98.62:43977)
0620 09:03:58.110236 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.110244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.111027 (+ 783us) server_negotiation.cc:407] Connection header received
0620 09:03:58.111961 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.111965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.112024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.112103 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:58.113363 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.113878 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.114978 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.115198 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.115309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.117857 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:58.117860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:58.117862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:58.117906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:58.119404 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.120001 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.120003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.120004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.120074 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.120495 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.120497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.120499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.120679 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:58.120799 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.121337 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.121479 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:03:58.122388 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.110431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43977 (local address 127.0.98.2:44919)
0620 09:03:58.110866 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.110909 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.111766 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.112308 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.112314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.112733 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:58.113228 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.113240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.113997 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.114005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.114857 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.114866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.115349 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.115353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.115512 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.116104 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:58.116121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:58.117693 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:58.119512 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:58.119518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:58.119533 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:58.119906 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:58.120156 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:58.120162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:58.120167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:58.120361 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:58.120794 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:58.120803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:58.121102 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.121261 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.121505 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260620 09:03:58.123512 19925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43977
I20260620 09:03:58.123775 19925 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:58.124385 19925 heartbeater.cc:511] Master 127.0.98.62:43977 requested a full tablet report, sending...
I20260620 09:03:58.125552 19600 ts_manager.cc:194] Registered new tserver with Master: 43cf0f8de34f47eaab9c87867cbc499c (127.0.98.2:40281)
I20260620 09:03:58.126103 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44919
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:03:58.144186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:03:58.251817 19930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:58.252064 19930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:58.252107 19930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:58.256120 19930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:58.256171 19930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:58.256278 19930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:58.261432 19930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:58.262403 19930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:58.263342 19930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:58.272403 19938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:58.272539 19930 server_base.cc:1034] running on GCE node
W20260620 09:03:58.272421 19935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:58.272414 19936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:58.273104 19930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:58.273729 19930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:58.274901 19930 hybrid_clock.cc:648] HybridClock initialized: now 1781946238274887 us; error 27 us; skew 500 ppm
Jun 20 09:03:58 dist-test-slave-scsf krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:03:58.277544 19930 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:03:58.278779 19930 webserver.cc:460] Webserver started at http://127.0.98.3:35143/ using document root <none> and password file <none>
I20260620 09:03:58.279397 19930 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:58.279461 19930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:58.279681 19930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:58.281812 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "5098e80b535245769083e12b1ee11d07"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "5ffefe7bbf1e710b813fca437b3f3f61"
server_key_iv: "91b8842960d01557102069b047c67b18"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.282399 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5098e80b535245769083e12b1ee11d07"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "5ffefe7bbf1e710b813fca437b3f3f61"
server_key_iv: "91b8842960d01557102069b047c67b18"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.285987 19930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:03:58.288338 19945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.289499 19930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:58.289633 19930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "5098e80b535245769083e12b1ee11d07"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "5ffefe7bbf1e710b813fca437b3f3f61"
server_key_iv: "91b8842960d01557102069b047c67b18"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.289776 19930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:58.308197 19930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:58.310818 19930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:58.311007 19930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:58.311583 19930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:58.312438 19930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:58.312497 19930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.312569 19930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:58.312616 19930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.321030 19930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33243
I20260620 09:03:58.321055 20058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33243 every 8 connection(s)
I20260620 09:03:58.321885 19930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:58.329792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 19930
I20260620 09:03:58.329914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:58.330255 392 external_mini_cluster.cc:1428] Setting key 75d4d45195345b21ab15e0695115154b
Jun 20 09:03:58 dist-test-slave-scsf krb5kdc[19555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:03:58.334363 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.323563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60589 (local address 127.0.98.62:43977)
0620 09:03:58.323730 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.323734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.324556 (+ 822us) server_negotiation.cc:407] Connection header received
0620 09:03:58.325386 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.325389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.325436 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.325508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:58.326569 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.327090 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.328152 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.328344 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.328413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.331012 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:58.331016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:58.331019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:58.331050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:58.332700 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.333137 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.333139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.333140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.333209 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.333518 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.333520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.333521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.333688 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:58.333759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.334169 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.334251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 09:03:58.335106 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.323999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43977 (local address 127.0.98.3:60589)
0620 09:03:58.324422 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.324476 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.325205 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.325661 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.325667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.325998 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:58.326452 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.326465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.327207 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.327209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.328058 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.328066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.328506 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.328513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.328657 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.329213 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:03:58.329228 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:58.330882 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:58.332794 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:58.332799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:58.332809 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:58.333052 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:58.333282 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:03:58.333284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:03:58.333285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:03:58.333437 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:03:58.333766 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:58.333770 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:58.333976 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.334122 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.334349 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260620 09:03:58.336190 20059 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43977
I20260620 09:03:58.336423 20059 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:58.336894 20059 heartbeater.cc:511] Master 127.0.98.62:43977 requested a full tablet report, sending...
I20260620 09:03:58.337932 19600 ts_manager.cc:194] Registered new tserver with Master: 5098e80b535245769083e12b1ee11d07 (127.0.98.3:33243)
I20260620 09:03:58.338442 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60589
I20260620 09:03:58.343688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:58.352880 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.345525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.0.98.62:43977)
0620 09:03:58.345672 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.345676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.345828 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:03:58.345917 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.345919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.345954 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.346044 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:58.346772 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.347283 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.348046 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.348253 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.348329 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.349587 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:58.349590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:58.349591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:03:58.349611 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:58.351317 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.351700 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.351702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.351703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.351775 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:03:58.351996 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:03:58.351998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:03:58.351999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:03:58.352192 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:58.352292 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.352574 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.352665 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 09:03:58.355580 19600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:58.358753 19600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:58.373028 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.367146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40281 (local address 127.0.0.1:49074)
0620 09:03:58.367660 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.367673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.367752 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.369121 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.369126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.369144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:58.369228 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.369235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.370574 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.370577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.371594 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.371603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.372709 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.372715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.372806 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.372825 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.372880 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.372931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":14,"thread_start_us":108,"threads_started":1}
I20260620 09:03:58.373137 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.367726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.0.98.2:40281)
0620 09:03:58.368713 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.368729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.368754 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:03:58.368825 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.368830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.368962 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.369367 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:58.369536 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.370419 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.371720 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.372541 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.372698 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.372850 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.372906 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.372989 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":156,"threads_started":1}
I20260620 09:03:58.373422 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.368130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.3:33243)
0620 09:03:58.368791 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.368803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.368826 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:03:58.368897 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.368901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.369088 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.369509 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:58.369663 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.370605 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.371784 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.372308 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.372422 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.372604 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.373144 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.373255 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":127,"threads_started":1}
I20260620 09:03:58.373029 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.367721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.0.1:51510)
0620 09:03:58.368350 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.368363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.368445 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.369242 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.369246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.369264 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:58.369365 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.369370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.370749 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.370752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.371655 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.371665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.372433 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.372437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.372588 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.372637 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.372746 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.372867 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260620 09:03:58.374390 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.367115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.1:38283)
0620 09:03:58.367988 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.368004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.368026 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:03:58.368101 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.368107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.368316 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.368512 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:58.368790 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.369994 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.372755 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.373556 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.373658 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.373800 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.374129 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.374244 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":74,"threads_started":1}
I20260620 09:03:58.374754 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.366956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38283 (local address 127.0.0.1:47346)
0620 09:03:58.367326 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.367376 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.367494 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.368486 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.368495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.368519 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:58.368670 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.368677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.370132 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.370135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.372624 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.372636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.373863 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.373868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.373967 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.373991 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.374528 (+ 537us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.374607 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":33,"thread_start_us":90,"threads_started":1}
I20260620 09:03:58.376492 19993 tablet_service.cc:1467] Processing CreateTablet for tablet 5802a5d734644b8f836da3c293bc03eb (DEFAULT_TABLE table=test-table [id=9fff1aea6a7b4c3e8f35f854ac8092d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:58.376493 19859 tablet_service.cc:1467] Processing CreateTablet for tablet 5802a5d734644b8f836da3c293bc03eb (DEFAULT_TABLE table=test-table [id=9fff1aea6a7b4c3e8f35f854ac8092d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:58.376964 19725 tablet_service.cc:1467] Processing CreateTablet for tablet 5802a5d734644b8f836da3c293bc03eb (DEFAULT_TABLE table=test-table [id=9fff1aea6a7b4c3e8f35f854ac8092d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:58.377611 19859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5802a5d734644b8f836da3c293bc03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:58.377611 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5802a5d734644b8f836da3c293bc03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:58.377926 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5802a5d734644b8f836da3c293bc03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:58.383885 20076 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Bootstrap starting.
I20260620 09:03:58.385458 20077 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Bootstrap starting.
I20260620 09:03:58.386734 20076 tablet_bootstrap.cc:654] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:58.387979 20077 tablet_bootstrap.cc:654] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:58.388026 20076 log.cc:824] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:58.388379 20078 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Bootstrap starting.
I20260620 09:03:58.388832 20077 log.cc:824] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:58.390249 20076 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: No bootstrap required, opened a new log
I20260620 09:03:58.390452 20076 ts_tablet_manager.cc:1397] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 09:03:58.390542 20078 tablet_bootstrap.cc:654] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:58.390811 20077 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: No bootstrap required, opened a new log
I20260620 09:03:58.391031 20077 ts_tablet_manager.cc:1397] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:03:58.391705 20078 log.cc:824] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:58.393741 20078 tablet_bootstrap.cc:492] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: No bootstrap required, opened a new log
I20260620 09:03:58.393920 20078 ts_tablet_manager.cc:1397] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:03:58.394012 20076 raft_consensus.cc:348] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.394276 20076 raft_consensus.cc:374] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:58.394331 20076 raft_consensus.cc:729] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bada63af68f94c32bd342853af94f86c, State: Initialized, Role: FOLLOWER
I20260620 09:03:58.394767 20076 consensus_queue.cc:260] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.395053 20077 raft_consensus.cc:348] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.395315 20077 raft_consensus.cc:374] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:58.395361 20077 raft_consensus.cc:729] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cf0f8de34f47eaab9c87867cbc499c, State: Initialized, Role: FOLLOWER
I20260620 09:03:58.395362 19791 heartbeater.cc:503] Master 127.0.98.62:43977 was elected leader, sending a full tablet report...
I20260620 09:03:58.395783 20076 ts_tablet_manager.cc:1428] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 09:03:58.395764 20077 consensus_queue.cc:260] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.396377 19925 heartbeater.cc:503] Master 127.0.98.62:43977 was elected leader, sending a full tablet report...
I20260620 09:03:58.396744 20077 ts_tablet_manager.cc:1428] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:03:58.397557 20078 raft_consensus.cc:348] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.397743 20078 raft_consensus.cc:374] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:58.397774 20078 raft_consensus.cc:729] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5098e80b535245769083e12b1ee11d07, State: Initialized, Role: FOLLOWER
I20260620 09:03:58.398125 20078 consensus_queue.cc:260] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.398743 20059 heartbeater.cc:503] Master 127.0.98.62:43977 was elected leader, sending a full tablet report...
I20260620 09:03:58.399003 20078 ts_tablet_manager.cc:1428] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260620 09:03:58.401336 19792 tablet.cc:2369] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:58.524138 20083 raft_consensus.cc:482] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:58.524374 20083 raft_consensus.cc:504] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.525568 20083 leader_election.cc:288] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bada63af68f94c32bd342853af94f86c (127.0.98.1:38283), 5098e80b535245769083e12b1ee11d07 (127.0.98.3:33243)
I20260620 09:03:58.529191 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.525936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41597 (local address 127.0.98.1:38283)
0620 09:03:58.526114 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.526119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.526136 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:03:58.526200 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.526206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.526269 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.526357 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:58.526698 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.527197 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.528234 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.528646 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.528761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.528826 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.528998 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.529070 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:03:58.529191 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.525829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38283 (local address 127.0.98.2:41597)
0620 09:03:58.525974 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.525993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.526107 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.526430 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.526433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.526455 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:58.526558 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.526563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.527309 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.527313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.528103 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.528112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.528762 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.528765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.528871 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.528895 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.528982 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.529054 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 09:03:58.529770 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.526272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35179 (local address 127.0.98.3:33243)
0620 09:03:58.526407 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.526411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.526578 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:03:58.526720 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.526723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.526773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.526853 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:58.527105 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.527587 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.528701 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.529107 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.529235 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.529304 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.529622 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.529684 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:03:58.529778 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.526129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33243 (local address 127.0.98.2:35179)
0620 09:03:58.526490 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.526501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:58.526611 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:58.526865 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:58.526868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:58.526882 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:58.527002 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.527007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.527716 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.527719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.528574 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:58.528583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.529380 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.529385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:58.529474 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.529496 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:03:58.529576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:03:58.529642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":144,"threads_started":1}
I20260620 09:03:58.529852 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5802a5d734644b8f836da3c293bc03eb" candidate_uuid: "43cf0f8de34f47eaab9c87867cbc499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bada63af68f94c32bd342853af94f86c" is_pre_election: true
I20260620 09:03:58.530200 19745 raft_consensus.cc:2454] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43cf0f8de34f47eaab9c87867cbc499c in term 0.
I20260620 09:03:58.530416 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5802a5d734644b8f836da3c293bc03eb" candidate_uuid: "43cf0f8de34f47eaab9c87867cbc499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5098e80b535245769083e12b1ee11d07" is_pre_election: true
I20260620 09:03:58.530720 20013 raft_consensus.cc:2454] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43cf0f8de34f47eaab9c87867cbc499c in term 0.
I20260620 09:03:58.530710 19815 leader_election.cc:302] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cf0f8de34f47eaab9c87867cbc499c, bada63af68f94c32bd342853af94f86c; no voters:
I20260620 09:03:58.530980 20083 raft_consensus.cc:2781] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:58.531072 20083 raft_consensus.cc:482] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:58.531127 20083 raft_consensus.cc:3037] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:58.532152 20083 raft_consensus.cc:504] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.532536 20083 leader_election.cc:288] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [CANDIDATE]: Term 1 election: Requested vote from peers bada63af68f94c32bd342853af94f86c (127.0.98.1:38283), 5098e80b535245769083e12b1ee11d07 (127.0.98.3:33243)
I20260620 09:03:58.532856 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5802a5d734644b8f836da3c293bc03eb" candidate_uuid: "43cf0f8de34f47eaab9c87867cbc499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bada63af68f94c32bd342853af94f86c"
I20260620 09:03:58.532905 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5802a5d734644b8f836da3c293bc03eb" candidate_uuid: "43cf0f8de34f47eaab9c87867cbc499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5098e80b535245769083e12b1ee11d07"
I20260620 09:03:58.532998 19745 raft_consensus.cc:3037] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:58.533021 20013 raft_consensus.cc:3037] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:58.534029 19745 raft_consensus.cc:2454] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43cf0f8de34f47eaab9c87867cbc499c in term 1.
I20260620 09:03:58.534065 20013 raft_consensus.cc:2454] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43cf0f8de34f47eaab9c87867cbc499c in term 1.
I20260620 09:03:58.534330 19815 leader_election.cc:302] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cf0f8de34f47eaab9c87867cbc499c, bada63af68f94c32bd342853af94f86c; no voters:
I20260620 09:03:58.534507 20083 raft_consensus.cc:2781] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:58.534729 20083 raft_consensus.cc:686] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 1 LEADER]: Becoming Leader. State: Replica: 43cf0f8de34f47eaab9c87867cbc499c, State: Running, Role: LEADER
I20260620 09:03:58.535106 20083 consensus_queue.cc:237] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } }
I20260620 09:03:58.538232 19599 catalog_manager.cc:5495] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c reported cstate change: term changed from 0 to 1, leader changed from <none> to 43cf0f8de34f47eaab9c87867cbc499c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "43cf0f8de34f47eaab9c87867cbc499c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43cf0f8de34f47eaab9c87867cbc499c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 } health_report { overall_health: UNKNOWN } } }
I20260620 09:03:58.563848 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.560764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.0.98.2:40281)
0620 09:03:58.560885 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.560889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.560901 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:03:58.560994 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.560996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.561050 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.561142 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:58.561341 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.561837 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.562605 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.562817 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.562888 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.563130 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:58.563542 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:58.563629 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.563694 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.563742 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:03:58.569612 19745 raft_consensus.cc:1261] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 1 FOLLOWER]: Refusing update from remote peer 43cf0f8de34f47eaab9c87867cbc499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:58.569617 20013 raft_consensus.cc:1261] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 1 FOLLOWER]: Refusing update from remote peer 43cf0f8de34f47eaab9c87867cbc499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:58.570304 20086 consensus_queue.cc:1035] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bada63af68f94c32bd342853af94f86c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:58.570472 20083 consensus_queue.cc:1035] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5098e80b535245769083e12b1ee11d07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:03:58.572711 20060 tablet.cc:2369] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:03:58.581696 20091 mvcc.cc:205] Tried to move back new op lower bound from 7298851793173348352 to 7298851793042313216. Current Snapshot: MvccSnapshot[applied={T|T < 7298851793173348352 or (T in {7298851793173348352})}]
I20260620 09:03:58.586575 19598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:58.586723 19598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:58.589111 19598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c98fb68f50bb4f52bb293d7a828076f8: Sending DeleteTablet for 3 replicas of tablet 5802a5d734644b8f836da3c293bc03eb
I20260620 09:03:58.589942 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet 5802a5d734644b8f836da3c293bc03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51510
I20260620 09:03:58.590003 19859 tablet_service.cc:1514] Processing DeleteTablet for tablet 5802a5d734644b8f836da3c293bc03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49074
I20260620 09:03:58.590045 19725 tablet_service.cc:1514] Processing DeleteTablet for tablet 5802a5d734644b8f836da3c293bc03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47346
I20260620 09:03:58.590549 20101 tablet_replica.cc:330] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: stopping tablet replica
I20260620 09:03:58.590662 20100 tablet_replica.cc:330] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c: stopping tablet replica
I20260620 09:03:58.590855 20101 raft_consensus.cc:2229] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:58.590940 20100 raft_consensus.cc:2229] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:58.591187 20101 raft_consensus.cc:2258] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:58.591253 20100 raft_consensus.cc:2258] T 5802a5d734644b8f836da3c293bc03eb P bada63af68f94c32bd342853af94f86c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:58.591459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19662
I20260620 09:03:58.591612 20099 tablet_replica.cc:330] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: stopping tablet replica
I20260620 09:03:58.591881 20099 raft_consensus.cc:2229] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:03:58.592228 20099 raft_consensus.cc:2258] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:58.593532 20101 ts_tablet_manager.cc:1905] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:58.594851 20099 ts_tablet_manager.cc:1905] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:58.596279 20101 ts_tablet_manager.cc:1918] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:58.596372 20101 log.cc:1196] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/5802a5d734644b8f836da3c293bc03eb
I20260620 09:03:58.596740 20101 ts_tablet_manager.cc:1939] T 5802a5d734644b8f836da3c293bc03eb P 5098e80b535245769083e12b1ee11d07: Deleting consensus metadata
I20260620 09:03:58.597971 19587 catalog_manager.cc:4841] TS 5098e80b535245769083e12b1ee11d07 (127.0.98.3:33243): tablet 5802a5d734644b8f836da3c293bc03eb (table test-table [id=9fff1aea6a7b4c3e8f35f854ac8092d2]) successfully deleted
I20260620 09:03:58.598125 20099 ts_tablet_manager.cc:1918] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:58.598225 20099 log.cc:1196] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/5802a5d734644b8f836da3c293bc03eb
I20260620 09:03:58.598517 20099 ts_tablet_manager.cc:1939] T 5802a5d734644b8f836da3c293bc03eb P 43cf0f8de34f47eaab9c87867cbc499c: Deleting consensus metadata
I20260620 09:03:58.599633 19584 catalog_manager.cc:4841] TS 43cf0f8de34f47eaab9c87867cbc499c (127.0.98.2:40281): tablet 5802a5d734644b8f836da3c293bc03eb (table test-table [id=9fff1aea6a7b4c3e8f35f854ac8092d2]) successfully deleted
W20260620 09:03:58.602022 19587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38283 (error 108)
I20260620 09:03:58.602859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19796
I20260620 09:03:58.603098 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.602760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38283 (local address 127.0.0.1:47354)
0620 09:03:58.602911 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:58.602992 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 09:03:58.608496 19587 catalog_manager.cc:4584] TS bada63af68f94c32bd342853af94f86c (127.0.98.1:38283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5802a5d734644b8f836da3c293bc03eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38283: connect: Connection refused (error 111)
I20260620 09:03:58.608657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19930
I20260620 09:03:58.613932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 19568
2026-06-20T09:03:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T09:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:03:58Z Disabled control of system clock
I20260620 09:03:58.643844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36087
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:58.762621 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:58.762876 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:58.762912 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:58.767098 20106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:03:58.767163 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:58.767179 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:58.767196 20106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:03:58.767211 20106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:03:58.771982 20106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:58.773402 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:58.774765 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:58.783283 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:58.783280 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:58.783437 20106 server_base.cc:1034] running on GCE node
W20260620 09:03:58.783615 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:58.784191 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:58.785281 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:58.786464 20106 hybrid_clock.cc:648] HybridClock initialized: now 1781946238786433 us; error 45 us; skew 500 ppm
I20260620 09:03:58.788250 20106 webserver.cc:460] Webserver started at http://127.0.98.62:42129/ using document root <none> and password file <none>
I20260620 09:03:58.788853 20106 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:58.788941 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:58.789175 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:58.791455 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "e8ec40f5039e48ac81c81f733fd5533d"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "7de1aaed389e491a1982b3406e08331c"
server_key_iv: "7755739d0965acbea03dba1a6442f515"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.792039 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8ec40f5039e48ac81c81f733fd5533d"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "7de1aaed389e491a1982b3406e08331c"
server_key_iv: "7755739d0965acbea03dba1a6442f515"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.795476 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:03:58.797756 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:58.798986 20106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:58.799125 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "e8ec40f5039e48ac81c81f733fd5533d"
format_stamp: "Formatted at 2026-06-20 09:03:58 on dist-test-slave-scsf"
server_key: "7de1aaed389e491a1982b3406e08331c"
server_key_iv: "7755739d0965acbea03dba1a6442f515"
server_key_version: "encryptionkey@0"
I20260620 09:03:58.799230 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:58.826172 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:58.826895 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:58.827095 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:58.834146 20106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36087
I20260620 09:03:58.834182 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36087 every 8 connection(s)
I20260620 09:03:58.835111 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:03:58.838219 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:58.839648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20106
I20260620 09:03:58.839835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:03:58.840152 392 external_mini_cluster.cc:1428] Setting key 57cb80c712b4633033a8996a44221936
I20260620 09:03:58.844393 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: Bootstrap starting.
I20260620 09:03:58.847090 20173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:58.848026 20173 log.cc:824] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:58.850644 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: No bootstrap required, opened a new log
I20260620 09:03:58.850991 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:58.841583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.0.98.62:36087)
0620 09:03:58.842238 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:58.842253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:58.842289 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:03:58.842945 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:58.842961 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:58.843348 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:58.843707 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:58.844136 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.845174 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.846571 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:58.847335 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:58.847509 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:58.848589 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:58.848596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:58.848613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:58.849165 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:58.849245 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:58.849274 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:58.849380 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:58.849588 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 09:03:58.849918 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":123,"threads_started":1}
I20260620 09:03:58.854736 20173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260620 09:03:58.854981 20173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:58.855060 20173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ec40f5039e48ac81c81f733fd5533d, State: Initialized, Role: FOLLOWER
I20260620 09:03:58.855444 20173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260620 09:03:58.855590 20173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:03:58.855701 20173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:03:58.855798 20173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:58.856956 20173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260620 09:03:58.857290 20173 leader_election.cc:302] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8ec40f5039e48ac81c81f733fd5533d; no voters:
I20260620 09:03:58.857604 20173 leader_election.cc:288] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:03:58.857715 20178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:58.857945 20178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [term 1 LEADER]: Becoming Leader. State: Replica: e8ec40f5039e48ac81c81f733fd5533d, State: Running, Role: LEADER
I20260620 09:03:58.858253 20178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260620 09:03:58.858801 20173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:03:58.859723 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8ec40f5039e48ac81c81f733fd5533d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } } }
I20260620 09:03:58.860141 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:58.860033 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8ec40f5039e48ac81c81f733fd5533d. Latest consensus state: current_term: 1 leader_uuid: "e8ec40f5039e48ac81c81f733fd5533d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ec40f5039e48ac81c81f733fd5533d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } } }
I20260620 09:03:58.860507 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d [sys.catalog]: This master's current role is: LEADER
I20260620 09:03:58.860620 20187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:03:58.863123 20187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:03:58.868415 20187 catalog_manager.cc:1347] Generated new cluster ID: b064fcf7fe60484d84ba9830795ef797
I20260620 09:03:58.868511 20187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:03:58.878377 20187 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:03:58.879271 20187 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:03:58.887745 20187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: Generated new TSK 0
I20260620 09:03:58.888407 20187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:03:58.893043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:03:59.007501 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:59.007790 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:59.007862 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:59.011833 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:59.011924 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:59.012013 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:03:59.016806 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:59.017895 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:59.019306 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:59.028671 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:59.028625 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:59.028648 20197 server_base.cc:1034] running on GCE node
W20260620 09:03:59.028861 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:59.029338 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:59.029942 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:59.031137 20197 hybrid_clock.cc:648] HybridClock initialized: now 1781946239031117 us; error 31 us; skew 500 ppm
I20260620 09:03:59.032963 20197 webserver.cc:460] Webserver started at http://127.0.98.1:34895/ using document root <none> and password file <none>
I20260620 09:03:59.033520 20197 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:59.033609 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:59.033838 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:59.035851 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "085d6e91503cde5b4ff77b00c1bc4bd5"
server_key_iv: "15734bc2fce5915081517532ca208693"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.036412 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "085d6e91503cde5b4ff77b00c1bc4bd5"
server_key_iv: "15734bc2fce5915081517532ca208693"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.040298 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:03:59.042762 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.043919 20197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:03:59.044049 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal
uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "085d6e91503cde5b4ff77b00c1bc4bd5"
server_key_iv: "15734bc2fce5915081517532ca208693"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.044158 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:59.055794 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:59.056392 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:59.056582 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:59.057080 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:59.057917 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:59.057983 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.058045 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:59.058090 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.066840 20197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34865
I20260620 09:03:59.066864 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34865 every 8 connection(s)
I20260620 09:03:59.067672 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:03:59.069351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20197
I20260620 09:03:59.069492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:03:59.069844 392 external_mini_cluster.cc:1428] Setting key 227744bb7a16f47165dd512aeb9661ff
I20260620 09:03:59.071820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:59.077700 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.069470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49343 (local address 127.0.98.62:36087)
0620 09:03:59.069615 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.069620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.070783 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 09:03:59.072186 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.072189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.072242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.072318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:59.073227 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.073757 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.075473 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.075720 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.075805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.077187 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:59.077190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:59.077192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:59.077204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:59.077221 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:59.077229 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:59.077293 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.077511 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.077570 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:03:59.079037 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.069921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.1:49343)
0620 09:03:59.070634 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.070682 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.071950 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.072540 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.072554 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.072639 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:59.073089 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.073101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.073888 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.073891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.075361 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.075373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.075883 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.075890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.076125 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.076868 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:59.076872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:59.076932 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:59.077023 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:59.077319 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:59.077330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:59.077339 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.077511 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.077842 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260620 09:03:59.080358 20325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260620 09:03:59.080646 20325 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:59.081147 20325 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260620 09:03:59.082595 20137 ts_manager.cc:194] Registered new tserver with Master: 2f5c9d6d67c040a1ad42f1ef5b01fe14 (127.0.98.1:34865)
I20260620 09:03:59.083786 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49343
W20260620 09:03:59.179136 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:59.179414 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:59.179500 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:59.183490 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:59.183571 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:59.183658 20328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:03:59.188917 20328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:59.189913 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:59.191222 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:59.200369 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:59.200431 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:59.200377 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:59.200555 20328 server_base.cc:1034] running on GCE node
I20260620 09:03:59.201045 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:59.201638 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:59.202790 20328 hybrid_clock.cc:648] HybridClock initialized: now 1781946239202774 us; error 23 us; skew 500 ppm
I20260620 09:03:59.204758 20328 webserver.cc:460] Webserver started at http://127.0.98.2:46205/ using document root <none> and password file <none>
I20260620 09:03:59.205309 20328 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:59.205369 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:59.205605 20328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:59.207546 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data/instance:
uuid: "247d75d83dba40dfbc1d98f70f6dbcd3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "edd6b5098ef9c911446a4e4add7a1500"
server_key_iv: "a2088ce3691b9ebf80024d0ba607e301"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.208074 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal/instance:
uuid: "247d75d83dba40dfbc1d98f70f6dbcd3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "edd6b5098ef9c911446a4e4add7a1500"
server_key_iv: "a2088ce3691b9ebf80024d0ba607e301"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.211591 20328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:03:59.213863 20342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.214999 20328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:03:59.215167 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal
uuid: "247d75d83dba40dfbc1d98f70f6dbcd3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "edd6b5098ef9c911446a4e4add7a1500"
server_key_iv: "a2088ce3691b9ebf80024d0ba607e301"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.215299 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:59.226904 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:59.227573 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:59.227746 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:59.228295 20328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:59.229244 20328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:59.229312 20328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.229390 20328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:59.229434 20328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.237928 20328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34081
I20260620 09:03:59.237967 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34081 every 8 connection(s)
I20260620 09:03:59.238808 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:03:59.247332 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.240539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44823 (local address 127.0.98.62:36087)
0620 09:03:59.240697 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.240701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.241639 (+ 938us) server_negotiation.cc:407] Connection header received
0620 09:03:59.242999 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.243002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.243046 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.243118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:59.243952 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.244516 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.245584 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.245795 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.245868 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.246881 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:59.246884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:59.246886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:59.246897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:59.246935 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:59.246947 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:59.247023 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.247165 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.247223 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:03:59.248006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20328
I20260620 09:03:59.248183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:03:59.248348 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.240898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.2:44823)
0620 09:03:59.241493 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.241530 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.242817 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.243274 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.243280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.243357 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:59.243832 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.243844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.244632 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.244634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.245467 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.245476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.245916 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.245919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.246048 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.246576 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:59.246578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:59.246619 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:59.246719 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:59.247037 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:59.247043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:59.247047 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.247143 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.247336 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260620 09:03:59.248558 392 external_mini_cluster.cc:1428] Setting key c7fc9f23a4d3e33b6e406460f7503f2a
I20260620 09:03:59.249459 20456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260620 09:03:59.249670 20456 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:59.250247 20456 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260620 09:03:59.250555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:03:59.251282 20137 ts_manager.cc:194] Registered new tserver with Master: 247d75d83dba40dfbc1d98f70f6dbcd3 (127.0.98.2:34081)
I20260620 09:03:59.252063 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44823
W20260620 09:03:59.358317 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:03:59.358615 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:03:59.358682 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:03:59.363044 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:03:59.363124 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:03:59.363205 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:03:59.367945 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:03:59.369138 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:03:59.370489 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:03:59.380649 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:03:59.380678 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:59.380741 20459 server_base.cc:1034] running on GCE node
W20260620 09:03:59.380966 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:03:59.381388 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:03:59.382038 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:03:59.383246 20459 hybrid_clock.cc:648] HybridClock initialized: now 1781946239383221 us; error 38 us; skew 500 ppm
I20260620 09:03:59.385197 20459 webserver.cc:460] Webserver started at http://127.0.98.3:39147/ using document root <none> and password file <none>
I20260620 09:03:59.385795 20459 fs_manager.cc:359] Metadata directory not provided
I20260620 09:03:59.385895 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:03:59.386142 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:03:59.388528 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data/instance:
uuid: "b255120b2a014d7e9b8fb21c999842c3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "5be2eaef362a882428aa495ecd06eac2"
server_key_iv: "1260101e2c5d341e7e59bb3c37b570ae"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.389190 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b255120b2a014d7e9b8fb21c999842c3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "5be2eaef362a882428aa495ecd06eac2"
server_key_iv: "1260101e2c5d341e7e59bb3c37b570ae"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.393078 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:03:59.395520 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.396867 20459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:03:59.396982 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal
uuid: "b255120b2a014d7e9b8fb21c999842c3"
format_stamp: "Formatted at 2026-06-20 09:03:59 on dist-test-slave-scsf"
server_key: "5be2eaef362a882428aa495ecd06eac2"
server_key_iv: "1260101e2c5d341e7e59bb3c37b570ae"
server_key_version: "encryptionkey@0"
I20260620 09:03:59.397106 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:03:59.414770 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:03:59.415357 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:03:59.415504 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:03:59.415994 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:03:59.416941 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:03:59.416990 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.417029 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:03:59.417052 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:03:59.426263 20459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41355
I20260620 09:03:59.426340 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41355 every 8 connection(s)
I20260620 09:03:59.427203 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:03:59.436600 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.428939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38071 (local address 127.0.98.62:36087)
0620 09:03:59.429078 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.429082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.430092 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 09:03:59.431592 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.431596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.431656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.431744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:59.432724 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.433288 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.434492 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.434757 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.434879 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.436042 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:59.436046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:59.436049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:59.436071 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:59.436090 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:59.436098 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:59.436177 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.436353 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.436414 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:03:59.436689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20459
I20260620 09:03:59.436815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:03:59.437070 392 external_mini_cluster.cc:1428] Setting key 71c8c0c51c00a20e02806374e72cc0e8
I20260620 09:03:59.437594 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.429322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.3:38071)
0620 09:03:59.429940 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.429981 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.431365 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.431931 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.431938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.432022 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:03:59.432584 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.432599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.433427 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.433436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.434360 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.434370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.434913 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.434918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.435097 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.435793 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:03:59.435795 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:03:59.435832 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:03:59.435892 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:03:59.436191 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:03:59.436201 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:03:59.436213 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.436339 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.436685 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260620 09:03:59.438782 20587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260620 09:03:59.439021 20587 heartbeater.cc:464] Registering TS with master...
I20260620 09:03:59.439536 20587 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260620 09:03:59.440436 20137 ts_manager.cc:194] Registered new tserver with Master: b255120b2a014d7e9b8fb21c999842c3 (127.0.98.3:41355)
I20260620 09:03:59.440953 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38071
I20260620 09:03:59.450793 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:03:59.460402 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.452656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.0.98.62:36087)
0620 09:03:59.452816 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.452823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.453025 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:03:59.453144 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.453147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.453203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.453285 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:03:59.453568 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.454074 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.454980 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.455209 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.455318 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.459910 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:03:59.459913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:03:59.459915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:03:59.459926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:03:59.459942 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:03:59.459950 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:03:59.460027 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.460153 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.460251 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:03:59.463054 20137 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:03:59.466102 20137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:03:59.480222 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.474749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34865 (local address 127.0.0.1:51956)
0620 09:03:59.475027 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.475040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.475136 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.476095 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.476100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.476119 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:59.476207 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.476213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.477506 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.477509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.478647 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.478662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.479718 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.479722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.479821 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.479855 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.479967 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.480075 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260620 09:03:59.480230 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.475323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.98.1:34865)
0620 09:03:59.475635 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.475643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.475663 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:03:59.475752 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.475764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.475939 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.476092 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:59.476317 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.477375 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.478813 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.479583 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.479718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.479899 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.479992 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.480076 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260620 09:03:59.480784 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.475328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.98.3:41355)
0620 09:03:59.475767 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.475785 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.475802 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:03:59.475858 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.475861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.475995 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.476162 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:59.476519 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.477441 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.478944 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.479719 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.479820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.479976 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.480518 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.480633 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":152,"threads_started":1}
I20260620 09:03:59.480932 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.474173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.98.2:34081)
0620 09:03:59.474565 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.474579 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.474615 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:03:59.474857 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.474864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.475096 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.475257 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:59.475732 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.476953 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.478786 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.479719 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.479820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.479976 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.480669 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.480757 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":102,"threads_started":1}
I20260620 09:03:59.481328 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.474124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41355 (local address 127.0.0.1:36640)
0620 09:03:59.474742 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.474755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.474831 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.476293 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.476298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.476318 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:59.476387 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.476391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.477572 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.477576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.478819 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.478829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.480039 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.480042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.480239 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.480308 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.480372 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.480430 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":75,"thread_start_us":86,"threads_started":1}
I20260620 09:03:59.481426 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.474054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34081 (local address 127.0.0.1:42702)
0620 09:03:59.474489 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.474530 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.474649 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.475354 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.475362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.475388 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:59.475527 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.475535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.477094 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.477098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.478597 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.478615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.480039 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.480042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.480241 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.480530 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.481193 (+ 663us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.481266 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":410,"thread_start_us":86,"threads_started":1}
I20260620 09:03:59.483454 20259 tablet_service.cc:1467] Processing CreateTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 (DEFAULT_TABLE table=test-table [id=9cbbe934129148dfae710e7b2ee88ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:59.483790 20521 tablet_service.cc:1467] Processing CreateTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 (DEFAULT_TABLE table=test-table [id=9cbbe934129148dfae710e7b2ee88ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:59.484058 20390 tablet_service.cc:1467] Processing CreateTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 (DEFAULT_TABLE table=test-table [id=9cbbe934129148dfae710e7b2ee88ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:03:59.484678 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79b8e3d83a14db8a208fe8b7a2266b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:59.484782 20521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79b8e3d83a14db8a208fe8b7a2266b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:59.484983 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79b8e3d83a14db8a208fe8b7a2266b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:03:59.490710 20604 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Bootstrap starting.
I20260620 09:03:59.492512 20606 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Bootstrap starting.
I20260620 09:03:59.492754 20605 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: Bootstrap starting.
I20260620 09:03:59.492993 20604 tablet_bootstrap.cc:654] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:59.493685 20604 log.cc:824] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:59.494540 20605 tablet_bootstrap.cc:654] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:59.494540 20606 tablet_bootstrap.cc:654] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Neither blocks nor log segments found. Creating new log.
I20260620 09:03:59.495370 20606 log.cc:824] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:59.495370 20605 log.cc:824] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: Log is configured to *not* fsync() on all Append() calls
I20260620 09:03:59.495976 20604 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: No bootstrap required, opened a new log
I20260620 09:03:59.496203 20604 ts_tablet_manager.cc:1397] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:03:59.497244 20606 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: No bootstrap required, opened a new log
I20260620 09:03:59.497244 20605 tablet_bootstrap.cc:492] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: No bootstrap required, opened a new log
I20260620 09:03:59.497416 20606 ts_tablet_manager.cc:1397] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:03:59.497416 20605 ts_tablet_manager.cc:1397] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:03:59.499425 20604 raft_consensus.cc:348] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.499661 20604 raft_consensus.cc:374] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:59.499732 20604 raft_consensus.cc:729] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b255120b2a014d7e9b8fb21c999842c3, State: Initialized, Role: FOLLOWER
I20260620 09:03:59.500213 20604 consensus_queue.cc:260] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.500870 20587 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
I20260620 09:03:59.501250 20604 ts_tablet_manager.cc:1428] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 09:03:59.501746 20605 raft_consensus.cc:348] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.502072 20605 raft_consensus.cc:374] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:59.502171 20605 raft_consensus.cc:729] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5c9d6d67c040a1ad42f1ef5b01fe14, State: Initialized, Role: FOLLOWER
I20260620 09:03:59.502607 20605 consensus_queue.cc:260] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.502977 20606 raft_consensus.cc:348] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.503238 20606 raft_consensus.cc:374] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:03:59.503298 20606 raft_consensus.cc:729] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d75d83dba40dfbc1d98f70f6dbcd3, State: Initialized, Role: FOLLOWER
I20260620 09:03:59.503362 20325 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
I20260620 09:03:59.503624 20605 ts_tablet_manager.cc:1428] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:03:59.503703 20606 consensus_queue.cc:260] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.504344 20456 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
I20260620 09:03:59.504662 20606 ts_tablet_manager.cc:1428] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 09:03:59.508633 20610 raft_consensus.cc:482] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:03:59.508760 20610 raft_consensus.cc:504] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.509618 20610 leader_election.cc:288] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 247d75d83dba40dfbc1d98f70f6dbcd3 (127.0.98.2:34081), 2f5c9d6d67c040a1ad42f1ef5b01fe14 (127.0.98.1:34865)
I20260620 09:03:59.513281 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.510182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35245 (local address 127.0.98.2:34081)
0620 09:03:59.510285 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.510290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.510301 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:03:59.510341 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.510343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.510389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.510457 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:59.510803 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.511292 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512288 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.512760 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512874 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.512927 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.513102 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.513162 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:03:59.513312 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.509814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34081 (local address 127.0.98.3:35245)
0620 09:03:59.509930 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.509946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.510029 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.510571 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.510575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.510591 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:59.510692 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.510697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.511396 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.511401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.512159 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.512176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.512874 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.512974 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.513001 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.513088 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.513156 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:03:59.513362 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.509877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55509 (local address 127.0.98.1:34865)
0620 09:03:59.510160 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.510176 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.510294 (+ 118us) server_negotiation.cc:407] Connection header received
0620 09:03:59.510429 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.510432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.510470 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.510551 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:03:59.510873 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.511360 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512349 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.512818 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512932 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.512986 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.513176 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.513243 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260620 09:03:59.513590 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.509827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34865 (local address 127.0.98.3:55509)
0620 09:03:59.510187 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.510202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:03:59.510329 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:03:59.510637 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:03:59.510641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:03:59.510662 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:03:59.510765 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.510770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.511464 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.511470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.512229 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:03:59.512237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.512932 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.512936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:03:59.513032 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.513060 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:03:59.513151 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:03:59.513220 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260620 09:03:59.513868 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79b8e3d83a14db8a208fe8b7a2266b6" candidate_uuid: "b255120b2a014d7e9b8fb21c999842c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" is_pre_election: true
I20260620 09:03:59.514144 20410 raft_consensus.cc:2454] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b255120b2a014d7e9b8fb21c999842c3 in term 0.
I20260620 09:03:59.514129 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79b8e3d83a14db8a208fe8b7a2266b6" candidate_uuid: "b255120b2a014d7e9b8fb21c999842c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" is_pre_election: true
I20260620 09:03:59.514380 20279 raft_consensus.cc:2454] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b255120b2a014d7e9b8fb21c999842c3 in term 0.
I20260620 09:03:59.514595 20475 leader_election.cc:302] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d75d83dba40dfbc1d98f70f6dbcd3, b255120b2a014d7e9b8fb21c999842c3; no voters:
I20260620 09:03:59.514870 20610 raft_consensus.cc:2781] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:03:59.514981 20610 raft_consensus.cc:482] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:03:59.515039 20610 raft_consensus.cc:3037] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:59.516002 20610 raft_consensus.cc:504] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.516351 20610 leader_election.cc:288] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [CANDIDATE]: Term 1 election: Requested vote from peers 247d75d83dba40dfbc1d98f70f6dbcd3 (127.0.98.2:34081), 2f5c9d6d67c040a1ad42f1ef5b01fe14 (127.0.98.1:34865)
I20260620 09:03:59.516656 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79b8e3d83a14db8a208fe8b7a2266b6" candidate_uuid: "b255120b2a014d7e9b8fb21c999842c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14"
I20260620 09:03:59.516640 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79b8e3d83a14db8a208fe8b7a2266b6" candidate_uuid: "b255120b2a014d7e9b8fb21c999842c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3"
I20260620 09:03:59.516770 20410 raft_consensus.cc:3037] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:59.516775 20279 raft_consensus.cc:3037] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:03:59.517750 20410 raft_consensus.cc:2454] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b255120b2a014d7e9b8fb21c999842c3 in term 1.
I20260620 09:03:59.517751 20279 raft_consensus.cc:2454] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b255120b2a014d7e9b8fb21c999842c3 in term 1.
I20260620 09:03:59.518028 20475 leader_election.cc:302] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d75d83dba40dfbc1d98f70f6dbcd3, b255120b2a014d7e9b8fb21c999842c3; no voters:
I20260620 09:03:59.518225 20610 raft_consensus.cc:2781] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:03:59.518461 20610 raft_consensus.cc:686] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 1 LEADER]: Becoming Leader. State: Replica: b255120b2a014d7e9b8fb21c999842c3, State: Running, Role: LEADER
I20260620 09:03:59.518777 20610 consensus_queue.cc:237] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } }
I20260620 09:03:59.521878 20136 catalog_manager.cc:5495] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b255120b2a014d7e9b8fb21c999842c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b255120b2a014d7e9b8fb21c999842c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b255120b2a014d7e9b8fb21c999842c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } health_report { overall_health: UNKNOWN } } }
I20260620 09:03:59.540946 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.537887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.0.98.3:41355)
0620 09:03:59.538017 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:03:59.538020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:03:59.538033 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:03:59.538115 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:03:59.538118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:03:59.538166 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:03:59.538250 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:03:59.538424 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.538920 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.539694 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:03:59.539910 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:03:59.539988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:03:59.540231 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:03:59.540644 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:03:59.540733 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:03:59.540788 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 09:03:59.540828 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:03:59.546480 20279 raft_consensus.cc:1261] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 1 FOLLOWER]: Refusing update from remote peer b255120b2a014d7e9b8fb21c999842c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:59.546505 20410 raft_consensus.cc:1261] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 1 FOLLOWER]: Refusing update from remote peer b255120b2a014d7e9b8fb21c999842c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:03:59.547160 20614 consensus_queue.cc:1035] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5c9d6d67c040a1ad42f1ef5b01fe14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:59.547426 20610 consensus_queue.cc:1035] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247d75d83dba40dfbc1d98f70f6dbcd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:03:59.553357 20617 mvcc.cc:205] Tried to move back new op lower bound from 7298851797175992320 to 7298851797071679488. Current Snapshot: MvccSnapshot[applied={T|T < 7298851797175992320}]
I20260620 09:03:59.553920 20619 mvcc.cc:205] Tried to move back new op lower bound from 7298851797175992320 to 7298851797071679488. Current Snapshot: MvccSnapshot[applied={T|T < 7298851797175992320}]
I20260620 09:03:59.555526 20618 mvcc.cc:205] Tried to move back new op lower bound from 7298851797175992320 to 7298851797071679488. Current Snapshot: MvccSnapshot[applied={T|T < 7298851797175992320}]
I20260620 09:03:59.563004 20137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:59.563135 20137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:03:59.565464 20137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8ec40f5039e48ac81c81f733fd5533d: Sending DeleteTablet for 3 replicas of tablet c79b8e3d83a14db8a208fe8b7a2266b6
I20260620 09:03:59.566147 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:59 UTC) from {username='slave'} at 127.0.0.1:42702
I20260620 09:03:59.566306 20521 tablet_service.cc:1514] Processing DeleteTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:59 UTC) from {username='slave'} at 127.0.0.1:36640
I20260620 09:03:59.566687 20626 tablet_replica.cc:330] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: stopping tablet replica
I20260620 09:03:59.566941 20259 tablet_service.cc:1514] Processing DeleteTablet for tablet c79b8e3d83a14db8a208fe8b7a2266b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:03:59 UTC) from {username='slave'} at 127.0.0.1:51956
I20260620 09:03:59.567224 20626 raft_consensus.cc:2229] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:59.567332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20197
I20260620 09:03:59.567423 20628 tablet_replica.cc:330] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14: stopping tablet replica
I20260620 09:03:59.567694 20628 raft_consensus.cc:2229] T c79b8e3d83a14db8a208fe8b7a2266b6 P 2f5c9d6d67c040a1ad42f1ef5b01fe14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:03:59.567890 20626 raft_consensus.cc:2258] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:59.568013 20627 tablet_replica.cc:330] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: stopping tablet replica
I20260620 09:03:59.568306 20627 raft_consensus.cc:2229] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:03:59.568693 20627 raft_consensus.cc:2258] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:03:59.569802 20626 ts_tablet_manager.cc:1905] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:59.570056 20627 ts_tablet_manager.cc:1905] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:03:59.573246 20626 ts_tablet_manager.cc:1918] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:59.573382 20627 ts_tablet_manager.cc:1918] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:03:59.573412 20626 log.cc:1196] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-1/wal/wals/c79b8e3d83a14db8a208fe8b7a2266b6
I20260620 09:03:59.573480 20627 log.cc:1196] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Deleting WAL directory at /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781945987698834-392-0/minicluster-data/ts-2/wal/wals/c79b8e3d83a14db8a208fe8b7a2266b6
I20260620 09:03:59.573802 20627 ts_tablet_manager.cc:1939] T c79b8e3d83a14db8a208fe8b7a2266b6 P b255120b2a014d7e9b8fb21c999842c3: Deleting consensus metadata
I20260620 09:03:59.573953 20626 ts_tablet_manager.cc:1939] T c79b8e3d83a14db8a208fe8b7a2266b6 P 247d75d83dba40dfbc1d98f70f6dbcd3: Deleting consensus metadata
I20260620 09:03:59.574952 20123 catalog_manager.cc:4841] TS b255120b2a014d7e9b8fb21c999842c3 (127.0.98.3:41355): tablet c79b8e3d83a14db8a208fe8b7a2266b6 (table test-table [id=9cbbe934129148dfae710e7b2ee88ba1]) successfully deleted
I20260620 09:03:59.575655 20122 catalog_manager.cc:4841] TS 247d75d83dba40dfbc1d98f70f6dbcd3 (127.0.98.2:34081): tablet c79b8e3d83a14db8a208fe8b7a2266b6 (table test-table [id=9cbbe934129148dfae710e7b2ee88ba1]) successfully deleted
W20260620 09:03:59.577011 20123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34865 (error 108)
I20260620 09:03:59.577255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20328
I20260620 09:03:59.579614 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:03:59.579278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34865 (local address 127.0.0.1:51962)
0620 09:03:59.579430 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:03:59.579511 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 09:03:59.579842 20123 catalog_manager.cc:4584] TS 2f5c9d6d67c040a1ad42f1ef5b01fe14 (127.0.98.1:34865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c79b8e3d83a14db8a208fe8b7a2266b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34865: connect: Connection refused (error 111)
I20260620 09:03:59.583595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20459
I20260620 09:03:59.589123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20106
2026-06-20T09:03:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (973 ms)
[----------] 6 tests from EncryptionPolicyTest (10297 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:03:59 dist-test-slave-scsf krb5kdc[20632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:03:59 dist-test-slave-scsf krb5kdc[20632](info): set up 2 sockets
Jun 20 09:03:59 dist-test-slave-scsf krb5kdc[20632](info): commencing operation
krb5kdc: starting...
W20260620 09:04:00.624513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:00 dist-test-slave-scsf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:00.744400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:00.851511 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:00.851811 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:00.851882 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:00.855857 20645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:00.855976 20645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:00.856015 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:00.856072 20645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:00.856112 20645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:00.861415 20645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:00.862439 20645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:00.863469 20645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:00.871644 20653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:00.871692 20651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:00.871645 20650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:00.871843 20645 server_base.cc:1034] running on GCE node
I20260620 09:04:00.872548 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:00.873610 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:00.874804 20645 hybrid_clock.cc:648] HybridClock initialized: now 1781946240874799 us; error 35 us; skew 500 ppm
Jun 20 09:04:00 dist-test-slave-scsf krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:00.877447 20645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:00.878274 20645 webserver.cc:460] Webserver started at http://127.0.0.1:44757/ using document root <none> and password file <none>
I20260620 09:04:00.878849 20645 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:00.878943 20645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:00.879171 20645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:00.881243 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "e0a6ed788ba9427aaaa46d4066f1688b"
format_stamp: "Formatted at 2026-06-20 09:04:00 on dist-test-slave-scsf"
server_key: "4b376453c47c72012e963930785b68b7"
server_key_iv: "0cb7f673c9554bad7bef278f4eaa913b"
server_key_version: "encryptionkey@0"
I20260620 09:04:00.881809 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0a6ed788ba9427aaaa46d4066f1688b"
format_stamp: "Formatted at 2026-06-20 09:04:00 on dist-test-slave-scsf"
server_key: "4b376453c47c72012e963930785b68b7"
server_key_iv: "0cb7f673c9554bad7bef278f4eaa913b"
server_key_version: "encryptionkey@0"
I20260620 09:04:00.885351 20645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:04:00.887687 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:00.888949 20645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:00.889086 20645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "e0a6ed788ba9427aaaa46d4066f1688b"
format_stamp: "Formatted at 2026-06-20 09:04:00 on dist-test-slave-scsf"
server_key: "4b376453c47c72012e963930785b68b7"
server_key_iv: "0cb7f673c9554bad7bef278f4eaa913b"
server_key_version: "encryptionkey@0"
I20260620 09:04:00.889197 20645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:00.900139 20645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:00.902918 20645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:00.903122 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:00.909977 20645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44677
I20260620 09:04:00.909996 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44677 every 8 connection(s)
I20260620 09:04:00.911012 20645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:00.913744 20713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:00.919137 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b: Bootstrap starting.
I20260620 09:04:00.920106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20645
I20260620 09:04:00.920253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:00.920631 392 external_mini_cluster.cc:1428] Setting key 611d4e79ee56582b04bc131a5271429d
I20260620 09:04:00.922114 20713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:00.922868 20713 log.cc:824] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:00.925077 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b: No bootstrap required, opened a new log
Jun 20 09:04:00 dist-test-slave-scsf krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:04:00.929476 20713 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } }
I20260620 09:04:00.929739 20713 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:00.929816 20713 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a6ed788ba9427aaaa46d4066f1688b, State: Initialized, Role: FOLLOWER
I20260620 09:04:00.930294 20713 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } }
I20260620 09:04:00.930455 20713 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:00.930528 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:00.930636 20713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:00.931882 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } }
I20260620 09:04:00.932263 20713 leader_election.cc:302] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a6ed788ba9427aaaa46d4066f1688b; no voters:
I20260620 09:04:00.932613 20713 leader_election.cc:288] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:00.932715 20718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:00.932945 20718 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [term 1 LEADER]: Becoming Leader. State: Replica: e0a6ed788ba9427aaaa46d4066f1688b, State: Running, Role: LEADER
I20260620 09:04:00.933297 20718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } }
I20260620 09:04:00.933842 20713 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:00.934501 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a6ed788ba9427aaaa46d4066f1688b. Latest consensus state: current_term: 1 leader_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } } }
I20260620 09:04:00.934676 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:00.935038 20725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:00.935330 20719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a6ed788ba9427aaaa46d4066f1688b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44677 } } }
I20260620 09:04:00.937158 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:00.921755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48187 (local address 127.0.0.1:44677)
0620 09:04:00.922145 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:00.922166 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:00.922201 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:04:00.922895 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:00.922912 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:00.923337 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:00.923715 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:00.924597 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:00.925625 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:00.926791 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:00.927251 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:00.927368 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:00.930332 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:00.930338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:00.930353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:00.930384 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:00.932330 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:00.933047 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:00.933053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:00.933058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:00.933193 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:00.933589 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:00.933593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:00.933594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:00.934081 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:00.934286 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:00.935766 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0620 09:04:00.936046 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260620 09:04:00.938210 20725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:00.939302 20719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:00.940589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:00.943840 20725 catalog_manager.cc:1347] Generated new cluster ID: 722753820b19491e957c9a4ca5924ae1
I20260620 09:04:00.943897 20725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:00.951598 20725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:00.952612 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:00.942770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38725 (local address 127.0.0.1:44677)
0620 09:04:00.942867 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:00.942871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:00.943097 (+ 226us) server_negotiation.cc:407] Connection header received
0620 09:04:00.943229 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:00.943232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:00.943277 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:00.943352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:00.944122 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:00.945342 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:00.946531 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:00.946877 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:00.946955 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:00.948844 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:00.948848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:00.948850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:00.948876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:00.950601 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:00.951193 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:00.951199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:00.951202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:00.951287 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:00.951670 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:00.951673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:00.951675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:00.951866 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:00.952013 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:00.952314 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 09:04:00.952411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:04:00.952867 20725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:00.957543 20725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0a6ed788ba9427aaaa46d4066f1688b: Generated new TSK 0
I20260620 09:04:00.958281 20725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:00.965157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20645
W20260620 09:04:00.970374 20630 connection.cc:535] client connection to 127.0.0.1:44677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44677): Connection reset by peer (error 104)
2026-06-20T09:04:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:01 dist-test-slave-scsf krb5kdc[20744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:01 dist-test-slave-scsf krb5kdc[20744](info): set up 2 sockets
Jun 20 09:04:01 dist-test-slave-scsf krb5kdc[20744](info): commencing operation
krb5kdc: starting...
W20260620 09:04:02.000607 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:02 dist-test-slave-scsf krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:02.116029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:02.220999 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:02.221264 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:02.221305 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:02.225207 20757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:02.225258 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:02.225277 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:02.225293 20757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:02.225308 20757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:02.230324 20757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:02.231263 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:02.232197 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:02.240355 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:02.240325 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:02.240325 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:02.240582 20757 server_base.cc:1034] running on GCE node
I20260620 09:04:02.241302 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:02.242341 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:02.243546 20757 hybrid_clock.cc:648] HybridClock initialized: now 1781946242243525 us; error 43 us; skew 500 ppm
Jun 20 09:04:02 dist-test-slave-scsf krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:02.246189 20757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:02.246999 20757 webserver.cc:460] Webserver started at http://127.0.0.1:43751/ using document root <none> and password file <none>
I20260620 09:04:02.247562 20757 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:02.247658 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:02.247876 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:02.249891 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "c0acdc542a3d4ab4982153a8e977352c"
format_stamp: "Formatted at 2026-06-20 09:04:02 on dist-test-slave-scsf"
server_key: "1a683fb0ecf4a29d90c237a427c7ba0f"
server_key_iv: "70ffec4b63bf617cacabe30a475eb81e"
server_key_version: "encryptionkey@0"
I20260620 09:04:02.250442 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0acdc542a3d4ab4982153a8e977352c"
format_stamp: "Formatted at 2026-06-20 09:04:02 on dist-test-slave-scsf"
server_key: "1a683fb0ecf4a29d90c237a427c7ba0f"
server_key_iv: "70ffec4b63bf617cacabe30a475eb81e"
server_key_version: "encryptionkey@0"
I20260620 09:04:02.254098 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:04:02.256363 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:02.257545 20757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:02.257682 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "c0acdc542a3d4ab4982153a8e977352c"
format_stamp: "Formatted at 2026-06-20 09:04:02 on dist-test-slave-scsf"
server_key: "1a683fb0ecf4a29d90c237a427c7ba0f"
server_key_iv: "70ffec4b63bf617cacabe30a475eb81e"
server_key_version: "encryptionkey@0"
I20260620 09:04:02.257790 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:02.274590 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:02.277396 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:02.277611 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:02.284502 20757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39857
I20260620 09:04:02.284528 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39857 every 8 connection(s)
I20260620 09:04:02.285423 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:02.287988 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:02.291870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20757
I20260620 09:04:02.292011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:02.292279 392 external_mini_cluster.cc:1428] Setting key 3042159ac6de88b7bae81d8e0ded9025
I20260620 09:04:02.293650 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c: Bootstrap starting.
I20260620 09:04:02.296339 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:02.297224 20825 log.cc:824] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:02.299311 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c: No bootstrap required, opened a new log
I20260620 09:04:02.302495 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:02.293683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56137 (local address 127.0.0.1:39857)
0620 09:04:02.294120 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:02.294139 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:02.294178 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:04:02.295022 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:02.295043 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:02.295457 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:02.295834 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:02.296248 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:02.297745 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:02.298760 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:02.299440 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:02.299592 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:02.300412 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:02.300418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:02.300433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:02.300952 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:02.301018 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:02.301047 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:02.301184 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:02.301356 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:04:02.301486 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260620 09:04:02.303773 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } }
I20260620 09:04:02.303979 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:02.304039 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0acdc542a3d4ab4982153a8e977352c, State: Initialized, Role: FOLLOWER
I20260620 09:04:02.304543 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } }
I20260620 09:04:02.304726 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:02.304795 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:02.304922 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:02.305501 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:02.305955 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } }
I20260620 09:04:02.306300 20825 leader_election.cc:302] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0acdc542a3d4ab4982153a8e977352c; no voters:
I20260620 09:04:02.306614 20825 leader_election.cc:288] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:02.306737 20830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:02.306964 20830 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [term 1 LEADER]: Becoming Leader. State: Replica: c0acdc542a3d4ab4982153a8e977352c, State: Running, Role: LEADER
I20260620 09:04:02.307296 20830 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } }
I20260620 09:04:02.307865 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:02.308494 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0acdc542a3d4ab4982153a8e977352c. Latest consensus state: current_term: 1 leader_uuid: "c0acdc542a3d4ab4982153a8e977352c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } } }
I20260620 09:04:02.308620 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:02.308878 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0acdc542a3d4ab4982153a8e977352c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0acdc542a3d4ab4982153a8e977352c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39857 } } }
I20260620 09:04:02.308970 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:02.309036 20842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:02.312271 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:02.307861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56565 (local address 127.0.0.1:39857)
0620 09:04:02.307971 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:02.307975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:02.307989 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:04:02.308023 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:02.308026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:02.308069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:02.308153 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:02.308476 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:02.309249 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:02.310504 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:02.310835 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:02.310944 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:02.311758 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:02.311765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:02.311770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:02.311784 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:02.311815 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:02.311832 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:02.311922 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:02.312053 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 09:04:02.312085 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:04:02.312119 20842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:02.318392 20842 catalog_manager.cc:1347] Generated new cluster ID: 416e9bf31f3b458ca3ad3e99f300cca3
I20260620 09:04:02.318456 20842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:02.330502 20842 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:02.331651 20842 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:02.342372 20842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0acdc542a3d4ab4982153a8e977352c: Generated new TSK 0
I20260620 09:04:02.343043 20842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:02.346685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20757
W20260620 09:04:02.352109 20742 connection.cc:535] client connection to 127.0.0.1:39857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39857): Connection reset by peer (error 104)
2026-06-20T09:04:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:02 dist-test-slave-scsf krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:02 dist-test-slave-scsf krb5kdc[20856](info): set up 2 sockets
Jun 20 09:04:02 dist-test-slave-scsf krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260620 09:04:03.372738 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:03 dist-test-slave-scsf krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:03.491102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:03.599227 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:03.599471 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:03.599510 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:03.603714 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:03.603787 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:03.603806 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:03.603852 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:03.603883 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:03.610332 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:03.611406 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:03.612432 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:03.620594 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:03.620599 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:03.620594 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:03.620698 20869 server_base.cc:1034] running on GCE node
I20260620 09:04:03.621382 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:03.622406 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:03.623594 20869 hybrid_clock.cc:648] HybridClock initialized: now 1781946243623562 us; error 46 us; skew 500 ppm
Jun 20 09:04:03 dist-test-slave-scsf krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:03.626214 20869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:03.627097 20869 webserver.cc:460] Webserver started at http://127.0.0.1:36553/ using document root <none> and password file <none>
I20260620 09:04:03.627627 20869 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:03.627717 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:03.627934 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:03.629889 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "f0fe296d23ab435a8a6e0885ae63556b"
format_stamp: "Formatted at 2026-06-20 09:04:03 on dist-test-slave-scsf"
server_key: "878867945927512090b1a94bef5841ea"
server_key_iv: "0b6f515253a4773d175c753efa2831eb"
server_key_version: "encryptionkey@0"
I20260620 09:04:03.630416 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0fe296d23ab435a8a6e0885ae63556b"
format_stamp: "Formatted at 2026-06-20 09:04:03 on dist-test-slave-scsf"
server_key: "878867945927512090b1a94bef5841ea"
server_key_iv: "0b6f515253a4773d175c753efa2831eb"
server_key_version: "encryptionkey@0"
I20260620 09:04:03.633898 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:04:03.636178 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:03.637385 20869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:03.637529 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "f0fe296d23ab435a8a6e0885ae63556b"
format_stamp: "Formatted at 2026-06-20 09:04:03 on dist-test-slave-scsf"
server_key: "878867945927512090b1a94bef5841ea"
server_key_iv: "0b6f515253a4773d175c753efa2831eb"
server_key_version: "encryptionkey@0"
I20260620 09:04:03.637646 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:03.642107 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:03.644865 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:03.645048 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:03.651985 20869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37619
I20260620 09:04:03.652009 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37619 every 8 connection(s)
I20260620 09:04:03.652967 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:03.655604 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:03.657025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20869
I20260620 09:04:03.657161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:03.657475 392 external_mini_cluster.cc:1428] Setting key ada24dbe730d7b0aba9b8361c5726bc0
I20260620 09:04:03.661859 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: Bootstrap starting.
I20260620 09:04:03.662523 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:03.658861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55811 (local address 127.0.0.1:37619)
0620 09:04:03.659279 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:03.659294 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:03.659327 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:04:03.659951 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:03.659967 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:03.660230 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:03.660596 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:03.661357 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:03.661361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:03.661373 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:03.661383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:03.661433 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:03.661451 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:03.661593 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:03.661672 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:04:03.661822 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
I20260620 09:04:03.664218 20937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:03.665091 20937 log.cc:824] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:03.665314 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:03.667294 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: No bootstrap required, opened a new log
I20260620 09:04:03.669040 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:03.667430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46629 (local address 127.0.0.1:37619)
0620 09:04:03.667543 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:03.667548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:03.667716 (+ 168us) server_negotiation.cc:407] Connection header received
0620 09:04:03.667801 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:03.667804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:03.667834 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:03.667918 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:03.668672 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:03.668676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:03.668679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:03.668691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:03.668718 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:03.668732 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:03.668821 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:03.668885 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:04:03.668930 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:04:03.671324 20937 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260620 09:04:03.671525 20937 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:03.671564 20937 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fe296d23ab435a8a6e0885ae63556b, State: Initialized, Role: FOLLOWER
I20260620 09:04:03.671998 20937 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260620 09:04:03.672118 20937 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:03.672159 20937 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:03.672231 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:03.673074 20937 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260620 09:04:03.673353 20937 leader_election.cc:302] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fe296d23ab435a8a6e0885ae63556b; no voters:
I20260620 09:04:03.673583 20937 leader_election.cc:288] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:03.673712 20947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:03.673977 20947 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [term 1 LEADER]: Becoming Leader. State: Replica: f0fe296d23ab435a8a6e0885ae63556b, State: Running, Role: LEADER
I20260620 09:04:03.674381 20947 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260620 09:04:03.674978 20937 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:03.676180 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fe296d23ab435a8a6e0885ae63556b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } } }
I20260620 09:04:03.676180 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fe296d23ab435a8a6e0885ae63556b. Latest consensus state: current_term: 1 leader_uuid: "f0fe296d23ab435a8a6e0885ae63556b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe296d23ab435a8a6e0885ae63556b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } } }
I20260620 09:04:03.676419 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:03.676327 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b [sys.catalog]: This master's current role is: LEADER
W20260620 09:04:03.680660 20962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:03.680768 20962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:03.680846 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:03.681517 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:03.686599 20958 catalog_manager.cc:1347] Generated new cluster ID: 8fb539bd272b4530a87fb9a51a178c62
I20260620 09:04:03.686657 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:03.699568 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:03.699954 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:03.705899 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fe296d23ab435a8a6e0885ae63556b: Generated new TSK 0
I20260620 09:04:03.706650 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:03.712581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20869
2026-06-20T09:04:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:03 dist-test-slave-scsf krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:03 dist-test-slave-scsf krb5kdc[20968](info): set up 2 sockets
Jun 20 09:04:03 dist-test-slave-scsf krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260620 09:04:04.740710 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:04 dist-test-slave-scsf krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:04.854839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:04.961658 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:04.961942 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:04.962023 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:04.966041 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:04.966113 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:04.966157 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:04.966216 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:04.966250 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:04.971369 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:04.972328 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:04.973346 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:04.981267 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:04.981256 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:04.981498 20981 server_base.cc:1034] running on GCE node
W20260620 09:04:04.981280 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:04.982234 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:04.983194 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:04.984395 20981 hybrid_clock.cc:648] HybridClock initialized: now 1781946244984369 us; error 38 us; skew 500 ppm
Jun 20 09:04:04 dist-test-slave-scsf krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:04.991600 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:04.992511 20981 webserver.cc:460] Webserver started at http://127.0.0.1:37163/ using document root <none> and password file <none>
I20260620 09:04:04.993047 20981 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:04.993132 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:04.993371 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:04.995218 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "2cb86d4873e6436c8733758303ea8d38"
format_stamp: "Formatted at 2026-06-20 09:04:04 on dist-test-slave-scsf"
server_key: "0617ad42cc478ff30450212ebafd76fc"
server_key_iv: "aaa72e36fb60f9f744a0f6e33a9c5453"
server_key_version: "encryptionkey@0"
I20260620 09:04:04.995760 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cb86d4873e6436c8733758303ea8d38"
format_stamp: "Formatted at 2026-06-20 09:04:04 on dist-test-slave-scsf"
server_key: "0617ad42cc478ff30450212ebafd76fc"
server_key_iv: "aaa72e36fb60f9f744a0f6e33a9c5453"
server_key_version: "encryptionkey@0"
I20260620 09:04:04.999228 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:04:05.001612 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:05.003068 20981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:04:05.003259 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "2cb86d4873e6436c8733758303ea8d38"
format_stamp: "Formatted at 2026-06-20 09:04:04 on dist-test-slave-scsf"
server_key: "0617ad42cc478ff30450212ebafd76fc"
server_key_iv: "aaa72e36fb60f9f744a0f6e33a9c5453"
server_key_version: "encryptionkey@0"
I20260620 09:04:05.003376 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:05.018980 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:05.022109 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:05.022307 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:05.029101 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37857
I20260620 09:04:05.029124 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37857 every 8 connection(s)
I20260620 09:04:05.029990 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:05.030210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 20981
I20260620 09:04:05.030341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:05.030649 392 external_mini_cluster.cc:1428] Setting key 2c3d8768e66da5d92e7a0b0490d75cd6
I20260620 09:04:05.033010 21049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:05.039386 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: Bootstrap starting.
Jun 20 09:04:05 dist-test-slave-scsf krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:04:05.041491 21049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:05.042171 21049 log.cc:824] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:05.044401 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: No bootstrap required, opened a new log
I20260620 09:04:05.044978 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:05.032015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56887 (local address 127.0.0.1:37857)
0620 09:04:05.032403 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:05.032418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:05.032495 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:04:05.033149 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:05.033176 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:05.033497 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:05.033761 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:05.034663 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:05.035623 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:05.036835 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:05.037289 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:05.037410 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:05.040145 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:05.040150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:05.040161 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:05.040197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:05.041938 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:05.042385 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:05.042390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:05.042396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:05.042537 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:05.042859 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:05.042861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:05.042862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:05.043351 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:05.043521 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:05.043805 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 09:04:05.044071 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20260620 09:04:05.047530 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:05.048019 21049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260620 09:04:05.048221 21049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:05.048280 21049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb86d4873e6436c8733758303ea8d38, State: Initialized, Role: FOLLOWER
I20260620 09:04:05.048776 21049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260620 09:04:05.048935 21049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:05.048996 21049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:05.049085 21049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:05.050171 21049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260620 09:04:05.050534 21049 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb86d4873e6436c8733758303ea8d38; no voters:
I20260620 09:04:05.050868 21049 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:05.050967 21059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:05.051159 21059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb86d4873e6436c8733758303ea8d38, State: Running, Role: LEADER
I20260620 09:04:05.051492 21059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260620 09:04:05.052047 21049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:05.052614 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb86d4873e6436c8733758303ea8d38. Latest consensus state: current_term: 1 leader_uuid: "2cb86d4873e6436c8733758303ea8d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260620 09:04:05.052738 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:05.053359 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cb86d4873e6436c8733758303ea8d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb86d4873e6436c8733758303ea8d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260620 09:04:05.053445 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38 [sys.catalog]: This master's current role is: LEADER
W20260620 09:04:05.057523 21074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:05.057670 21074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:05.058204 21065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:05.058310 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:05.049497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53425 (local address 127.0.0.1:37857)
0620 09:04:05.049616 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:05.049619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:05.049776 (+ 157us) server_negotiation.cc:407] Connection header received
0620 09:04:05.049882 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:05.049884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:05.049916 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:05.049980 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:05.050805 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:05.051558 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:05.052777 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:05.053158 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:05.053274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:05.054947 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:05.054951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:05.054953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:05.054974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:05.056438 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:05.056920 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:05.056923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:05.056924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:05.057005 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:05.057316 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:05.057320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:05.057321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:05.057523 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:05.057640 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:05.057976 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 09:04:05.058097 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260620 09:04:05.059144 21065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:05.063880 21065 catalog_manager.cc:1347] Generated new cluster ID: a4163a4e421e4ae1968ac5d351c59016
I20260620 09:04:05.063941 21065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:05.083523 21065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:05.084499 21065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:05.089645 21065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cb86d4873e6436c8733758303ea8d38: Generated new TSK 0
I20260620 09:04:05.090274 21065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:05.100569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 20981
2026-06-20T09:04:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:05 dist-test-slave-scsf krb5kdc[21080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:05 dist-test-slave-scsf krb5kdc[21080](info): set up 2 sockets
Jun 20 09:04:05 dist-test-slave-scsf krb5kdc[21080](info): commencing operation
krb5kdc: starting...
W20260620 09:04:06.124696 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:06 dist-test-slave-scsf krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:06.241621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:06.346336 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:06.346572 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:06.346611 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:06.350528 21093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:06.350579 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:06.350597 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:06.350613 21093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:06.350628 21093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:06.356034 21093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:06.357107 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:06.358053 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:06.366219 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:06.366225 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:06.366261 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:06.366271 21093 server_base.cc:1034] running on GCE node
I20260620 09:04:06.367035 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:06.368110 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:06.369321 21093 hybrid_clock.cc:648] HybridClock initialized: now 1781946246369308 us; error 25 us; skew 500 ppm
Jun 20 09:04:06 dist-test-slave-scsf krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:06.371991 21093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:06.372893 21093 webserver.cc:460] Webserver started at http://127.0.0.1:36565/ using document root <none> and password file <none>
I20260620 09:04:06.373438 21093 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:06.373526 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:06.373759 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:06.375757 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "c721b9c9903c423e8c1affdc7a66bdcf"
format_stamp: "Formatted at 2026-06-20 09:04:06 on dist-test-slave-scsf"
server_key: "2e895dbae3b035b2bad3df4f4f6a7937"
server_key_iv: "c65298aa3453894e1af60aae65f70fa3"
server_key_version: "encryptionkey@0"
I20260620 09:04:06.376314 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "c721b9c9903c423e8c1affdc7a66bdcf"
format_stamp: "Formatted at 2026-06-20 09:04:06 on dist-test-slave-scsf"
server_key: "2e895dbae3b035b2bad3df4f4f6a7937"
server_key_iv: "c65298aa3453894e1af60aae65f70fa3"
server_key_version: "encryptionkey@0"
I20260620 09:04:06.379877 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:04:06.382221 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:06.383340 21093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:06.383477 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "c721b9c9903c423e8c1affdc7a66bdcf"
format_stamp: "Formatted at 2026-06-20 09:04:06 on dist-test-slave-scsf"
server_key: "2e895dbae3b035b2bad3df4f4f6a7937"
server_key_iv: "c65298aa3453894e1af60aae65f70fa3"
server_key_version: "encryptionkey@0"
I20260620 09:04:06.383584 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:06.397326 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:06.400027 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:06.400226 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:06.406827 21093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40529
I20260620 09:04:06.406838 21160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40529 every 8 connection(s)
I20260620 09:04:06.407814 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:06.410468 21161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:06.415477 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf: Bootstrap starting.
I20260620 09:04:06.417286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21093
I20260620 09:04:06.417433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:06.417672 392 external_mini_cluster.cc:1428] Setting key 04a37790c99a1f9890f9f5656540531d
I20260620 09:04:06.417956 21161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:06.418850 21161 log.cc:824] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:06.420858 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf: No bootstrap required, opened a new log
I20260620 09:04:06.424921 21161 raft_consensus.cc:348] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260620 09:04:06.425184 21161 raft_consensus.cc:374] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:06.425290 21161 raft_consensus.cc:729] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c721b9c9903c423e8c1affdc7a66bdcf, State: Initialized, Role: FOLLOWER
I20260620 09:04:06.425745 21161 consensus_queue.cc:260] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260620 09:04:06.425915 21161 raft_consensus.cc:388] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:06.426002 21161 raft_consensus.cc:482] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:06.426088 21161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:06.426286 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:06.418754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39893 (local address 127.0.0.1:40529)
0620 09:04:06.419143 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:06.419158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:06.419193 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:04:06.419822 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:06.419851 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:06.420158 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:06.420429 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:06.420867 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:06.421833 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:06.423065 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:06.423522 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:06.423631 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:06.424516 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:06.424521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:06.424533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:06.424901 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:06.424949 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:06.424970 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:06.425041 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:06.425234 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:04:06.425448 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260620 09:04:06.427158 21161 raft_consensus.cc:504] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260620 09:04:06.427475 21161 leader_election.cc:302] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c721b9c9903c423e8c1affdc7a66bdcf; no voters:
I20260620 09:04:06.427815 21161 leader_election.cc:288] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:06.427879 21166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:06.428085 21166 raft_consensus.cc:686] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [term 1 LEADER]: Becoming Leader. State: Replica: c721b9c9903c423e8c1affdc7a66bdcf, State: Running, Role: LEADER
I20260620 09:04:06.428444 21166 consensus_queue.cc:237] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260620 09:04:06.429062 21161 sys_catalog.cc:564] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:06.429313 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:06.430128 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } } }
I20260620 09:04:06.430308 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:06.430608 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c721b9c9903c423e8c1affdc7a66bdcf. Latest consensus state: current_term: 1 leader_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c721b9c9903c423e8c1affdc7a66bdcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } } }
I20260620 09:04:06.430738 21175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:06.430773 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:06.433877 21175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:06.436319 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:06.431800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44597 (local address 127.0.0.1:40529)
0620 09:04:06.431889 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:06.431893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:06.432329 (+ 436us) server_negotiation.cc:407] Connection header received
0620 09:04:06.432527 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:06.432529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:06.432578 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:06.432635 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:06.433006 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:06.433508 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:06.434756 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:06.434994 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:06.435101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:06.435899 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:06.435903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:06.435905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:06.435916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:06.435948 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:06.435958 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:06.436050 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:06.436137 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:04:06.436184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:04:06.440253 21175 catalog_manager.cc:1347] Generated new cluster ID: f412ac4de2eb401cbcc93e83784ecd89
I20260620 09:04:06.440328 21175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:06.451659 21175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:06.452612 21175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:06.457355 21175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c721b9c9903c423e8c1affdc7a66bdcf: Generated new TSK 0
I20260620 09:04:06.458264 21175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:06.469039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21093
2026-06-20T09:04:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:06 dist-test-slave-scsf krb5kdc[21192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:06 dist-test-slave-scsf krb5kdc[21192](info): set up 2 sockets
Jun 20 09:04:06 dist-test-slave-scsf krb5kdc[21192](info): commencing operation
krb5kdc: starting...
W20260620 09:04:07.504648 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:07 dist-test-slave-scsf krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:07.619321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:07.725186 21205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:07.725416 21205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:07.725455 21205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:07.729187 21205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:07.729234 21205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:07.729254 21205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:07.729269 21205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:07.729283 21205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:07.734372 21205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:07.735306 21205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:07.736238 21205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:04:07.744168 21205 server_base.cc:1034] running on GCE node
W20260620 09:04:07.744204 21210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:07.744189 21211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:07.744385 21213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:07.744899 21205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:07.745886 21205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:07.747066 21205 hybrid_clock.cc:648] HybridClock initialized: now 1781946247747042 us; error 36 us; skew 500 ppm
Jun 20 09:04:07 dist-test-slave-scsf krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:07.749634 21205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:07.750424 21205 webserver.cc:460] Webserver started at http://127.0.0.1:40077/ using document root <none> and password file <none>
I20260620 09:04:07.750928 21205 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:07.751016 21205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:07.751231 21205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:07.753218 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "56e3a8243f354bb8a349fd2777970614"
format_stamp: "Formatted at 2026-06-20 09:04:07 on dist-test-slave-scsf"
server_key: "ec5fc2bcc29fd725804f2a1954e253f3"
server_key_iv: "8f9fd7d78553befe1c9e518278adf51f"
server_key_version: "encryptionkey@0"
I20260620 09:04:07.753767 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "56e3a8243f354bb8a349fd2777970614"
format_stamp: "Formatted at 2026-06-20 09:04:07 on dist-test-slave-scsf"
server_key: "ec5fc2bcc29fd725804f2a1954e253f3"
server_key_iv: "8f9fd7d78553befe1c9e518278adf51f"
server_key_version: "encryptionkey@0"
I20260620 09:04:07.757267 21205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:04:07.759414 21220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:07.760521 21205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:07.760637 21205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "56e3a8243f354bb8a349fd2777970614"
format_stamp: "Formatted at 2026-06-20 09:04:07 on dist-test-slave-scsf"
server_key: "ec5fc2bcc29fd725804f2a1954e253f3"
server_key_iv: "8f9fd7d78553befe1c9e518278adf51f"
server_key_version: "encryptionkey@0"
I20260620 09:04:07.760731 21205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:07.765007 21205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:07.767930 21205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:07.768132 21205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:07.774502 21205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37667
I20260620 09:04:07.774533 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37667 every 8 connection(s)
I20260620 09:04:07.775333 21205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:07.777917 21273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:07.783056 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614: Bootstrap starting.
I20260620 09:04:07.784565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21205
I20260620 09:04:07.784703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:07.784976 392 external_mini_cluster.cc:1428] Setting key c675e896e8b5fd0faa6500337ec879d9
I20260620 09:04:07.785509 21273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:07.786388 21273 log.cc:824] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:07.788161 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614: No bootstrap required, opened a new log
I20260620 09:04:07.789598 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:07.785969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45467 (local address 127.0.0.1:37667)
0620 09:04:07.786441 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:07.786461 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:07.786498 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:04:07.787168 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:07.787185 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:07.787434 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:07.787772 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:07.788506 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:07.788511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:07.788525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:07.788542 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:07.788590 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:07.788608 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:07.788714 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:07.788823 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:04:07.788972 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260620 09:04:07.792393 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:07.792178 21273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260620 09:04:07.792506 21273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:07.792572 21273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e3a8243f354bb8a349fd2777970614, State: Initialized, Role: FOLLOWER
I20260620 09:04:07.792995 21273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260620 09:04:07.793118 21273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:07.793208 21273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:07.793305 21273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:07.794229 21273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260620 09:04:07.794615 21273 leader_election.cc:302] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e3a8243f354bb8a349fd2777970614; no voters:
I20260620 09:04:07.794943 21273 leader_election.cc:288] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:07.795030 21283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:07.795251 21283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [term 1 LEADER]: Becoming Leader. State: Replica: 56e3a8243f354bb8a349fd2777970614, State: Running, Role: LEADER
I20260620 09:04:07.795557 21283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260620 09:04:07.796276 21273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:07.796309 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:07.794450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45211 (local address 127.0.0.1:37667)
0620 09:04:07.794568 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:07.794572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:07.794781 (+ 209us) server_negotiation.cc:407] Connection header received
0620 09:04:07.794822 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:07.794824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:07.794889 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:07.794975 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:07.795785 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:07.795788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:07.795790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:07.795802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:07.795824 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:07.795837 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:07.795909 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:07.796168 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:04:07.796203 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:04:07.797436 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e3a8243f354bb8a349fd2777970614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
I20260620 09:04:07.797555 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:07.797833 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e3a8243f354bb8a349fd2777970614. Latest consensus state: current_term: 1 leader_uuid: "56e3a8243f354bb8a349fd2777970614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e3a8243f354bb8a349fd2777970614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
I20260620 09:04:07.797964 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:07.798022 21292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:07.801046 21292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:07.806551 21292 catalog_manager.cc:1347] Generated new cluster ID: cd36647c27994a96b35ee5ebc0dac212
I20260620 09:04:07.806646 21292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:07.813357 21292 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:07.813616 21292 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:07.819721 21292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e3a8243f354bb8a349fd2777970614: Generated new TSK 0
I20260620 09:04:07.820338 21292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:07.833353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21205
2026-06-20T09:04:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:07 dist-test-slave-scsf krb5kdc[21304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:07 dist-test-slave-scsf krb5kdc[21304](info): set up 2 sockets
Jun 20 09:04:07 dist-test-slave-scsf krb5kdc[21304](info): commencing operation
krb5kdc: starting...
W20260620 09:04:08.880776 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:08 dist-test-slave-scsf krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:08.997277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:09.103711 21317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:09.103955 21317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:09.103994 21317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:09.107795 21317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:09.107851 21317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:09.107869 21317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:09.107885 21317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:09.107899 21317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:09.113003 21317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:09.114111 21317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:09.115037 21317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:09.123052 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:09.123109 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:09.123257 21317 server_base.cc:1034] running on GCE node
W20260620 09:04:09.123049 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:09.123962 21317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:09.125078 21317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:09.126267 21317 hybrid_clock.cc:648] HybridClock initialized: now 1781946249126235 us; error 45 us; skew 500 ppm
Jun 20 09:04:09 dist-test-slave-scsf krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:09.128893 21317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:09.129710 21317 webserver.cc:460] Webserver started at http://127.0.0.1:42569/ using document root <none> and password file <none>
I20260620 09:04:09.130178 21317 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:09.130231 21317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:09.130398 21317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:09.132242 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "e5f109e1c09b464a98b7f6d5fbfd4395"
format_stamp: "Formatted at 2026-06-20 09:04:09 on dist-test-slave-scsf"
server_key: "08211dd8193c5736c2432644acbdea33"
server_key_iv: "55b0ad05c5e104c36184b05e2bd9111c"
server_key_version: "encryptionkey@0"
I20260620 09:04:09.132871 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5f109e1c09b464a98b7f6d5fbfd4395"
format_stamp: "Formatted at 2026-06-20 09:04:09 on dist-test-slave-scsf"
server_key: "08211dd8193c5736c2432644acbdea33"
server_key_iv: "55b0ad05c5e104c36184b05e2bd9111c"
server_key_version: "encryptionkey@0"
I20260620 09:04:09.136339 21317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:04:09.138753 21332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:09.139882 21317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:09.140017 21317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "e5f109e1c09b464a98b7f6d5fbfd4395"
format_stamp: "Formatted at 2026-06-20 09:04:09 on dist-test-slave-scsf"
server_key: "08211dd8193c5736c2432644acbdea33"
server_key_iv: "55b0ad05c5e104c36184b05e2bd9111c"
server_key_version: "encryptionkey@0"
I20260620 09:04:09.140115 21317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:09.179065 21317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:09.182032 21317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:09.182246 21317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:09.189579 21317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35835
I20260620 09:04:09.189597 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35835 every 8 connection(s)
I20260620 09:04:09.190517 21317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:09.193145 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:09.193660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21317
I20260620 09:04:09.193800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:09.194103 392 external_mini_cluster.cc:1428] Setting key 220b37f233167d1ce8690c6e8697c019
I20260620 09:04:09.199071 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395: Bootstrap starting.
I20260620 09:04:09.201261 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:09.202142 21385 log.cc:824] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:04:09 dist-test-slave-scsf krb5kdc[21304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:04:09.204588 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395: No bootstrap required, opened a new log
I20260620 09:04:09.207965 21385 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 09:04:09.208201 21385 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:09.208298 21385 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f109e1c09b464a98b7f6d5fbfd4395, State: Initialized, Role: FOLLOWER
I20260620 09:04:09.208802 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 09:04:09.208961 21385 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:09.209046 21385 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:09.209142 21385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:09.209358 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:09.195516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:45101 (local address 127.0.0.1:35835)
0620 09:04:09.195985 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:09.196000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:09.196037 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:04:09.196852 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:09.196871 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:09.197231 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:09.197587 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:09.198482 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:09.199597 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:09.200569 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:09.201066 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:09.201254 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:09.204733 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:09.204737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:09.204750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:09.204775 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:09.206301 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:09.206821 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:09.206826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:09.206830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:09.206931 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:09.207268 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:09.207272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:09.207274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:09.207633 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:09.207870 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:09.208126 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 09:04:09.208369 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":114,"threads_started":1}
I20260620 09:04:09.210278 21385 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 09:04:09.210637 21385 leader_election.cc:302] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f109e1c09b464a98b7f6d5fbfd4395; no voters:
I20260620 09:04:09.210963 21385 leader_election.cc:288] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:09.211099 21390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:09.211324 21390 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [term 1 LEADER]: Becoming Leader. State: Replica: e5f109e1c09b464a98b7f6d5fbfd4395, State: Running, Role: LEADER
I20260620 09:04:09.211716 21390 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } }
I20260620 09:04:09.212183 21385 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:09.212774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:09.213727 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f109e1c09b464a98b7f6d5fbfd4395. Latest consensus state: current_term: 1 leader_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } } }
I20260620 09:04:09.213727 21391 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f109e1c09b464a98b7f6d5fbfd4395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35835 } } }
I20260620 09:04:09.213886 21391 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:09.213886 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:09.214461 21405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:09.218098 21405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:09.224148 21405 catalog_manager.cc:1347] Generated new cluster ID: 622824b82bad48bbbcf7549eb0d5b43e
I20260620 09:04:09.224206 21405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:09.227026 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:09.215813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46235 (local address 127.0.0.1:35835)
0620 09:04:09.215933 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:09.215936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:09.216305 (+ 369us) server_negotiation.cc:407] Connection header received
0620 09:04:09.216512 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:09.216515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:09.216561 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:09.216611 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:09.217491 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:09.218213 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:09.219355 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:09.219676 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:09.219764 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:09.223046 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:09.223050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:09.223053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:09.223076 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:09.225219 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:09.225757 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:09.225762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:09.225764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:09.225850 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:09.226158 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:09.226163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:09.226164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:09.226367 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:09.226518 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:09.226766 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 09:04:09.226859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:04:09.233304 21405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:09.234408 21405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:09.245555 21405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f109e1c09b464a98b7f6d5fbfd4395: Generated new TSK 0
I20260620 09:04:09.246210 21405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:09.252208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21317
W20260620 09:04:09.256883 21302 connection.cc:535] client connection to 127.0.0.1:35835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35835): Connection reset by peer (error 104)
2026-06-20T09:04:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:09 dist-test-slave-scsf krb5kdc[21416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:09 dist-test-slave-scsf krb5kdc[21416](info): set up 2 sockets
Jun 20 09:04:09 dist-test-slave-scsf krb5kdc[21416](info): commencing operation
krb5kdc: starting...
W20260620 09:04:10.280795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:10 dist-test-slave-scsf krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:10.399574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:10.508409 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:10.508757 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:10.508878 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:10.512825 21429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:10.512914 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:10.512952 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:10.512989 21429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:10.513018 21429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:10.518481 21429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:10.519547 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:10.520612 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:10.528719 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:10.528841 21429 server_base.cc:1034] running on GCE node
W20260620 09:04:10.528788 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:10.528973 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:10.529567 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:10.530685 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:10.531872 21429 hybrid_clock.cc:648] HybridClock initialized: now 1781946250531841 us; error 46 us; skew 500 ppm
Jun 20 09:04:10 dist-test-slave-scsf krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:10.538616 21429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:10.539549 21429 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
I20260620 09:04:10.540112 21429 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:10.540203 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:10.540488 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:10.542514 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "b5bce866709141d89e83881b421fe69c"
format_stamp: "Formatted at 2026-06-20 09:04:10 on dist-test-slave-scsf"
server_key: "e250e4b3584865b59a26cbccdc4c19c5"
server_key_iv: "2ddfacea2cdf9b152a2bdd4cb9695a42"
server_key_version: "encryptionkey@0"
I20260620 09:04:10.543078 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5bce866709141d89e83881b421fe69c"
format_stamp: "Formatted at 2026-06-20 09:04:10 on dist-test-slave-scsf"
server_key: "e250e4b3584865b59a26cbccdc4c19c5"
server_key_iv: "2ddfacea2cdf9b152a2bdd4cb9695a42"
server_key_version: "encryptionkey@0"
I20260620 09:04:10.546743 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:04:10.549091 21444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:10.550264 21429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:10.550401 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "b5bce866709141d89e83881b421fe69c"
format_stamp: "Formatted at 2026-06-20 09:04:10 on dist-test-slave-scsf"
server_key: "e250e4b3584865b59a26cbccdc4c19c5"
server_key_iv: "2ddfacea2cdf9b152a2bdd4cb9695a42"
server_key_version: "encryptionkey@0"
I20260620 09:04:10.550509 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:10.568243 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:10.571748 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:10.571950 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:10.578969 21429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43371
I20260620 09:04:10.578996 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43371 every 8 connection(s)
I20260620 09:04:10.579885 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:10.582559 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:10.585391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21429
I20260620 09:04:10.585527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:10.585800 392 external_mini_cluster.cc:1428] Setting key c87ace9972624f9fb00ce1e6f66633ef
I20260620 09:04:10.588500 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c: Bootstrap starting.
I20260620 09:04:10.591123 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:10.592025 21497 log.cc:824] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:10.594126 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c: No bootstrap required, opened a new log
I20260620 09:04:10.595654 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:10.587175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:58961 (local address 127.0.0.1:43371)
0620 09:04:10.587587 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:10.587605 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:10.587646 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:04:10.588411 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:10.588432 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:10.588893 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:10.589237 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:10.589653 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:10.591138 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:10.592385 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:10.593014 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:10.593139 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:10.593976 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:10.593981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:10.593993 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:10.594407 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:10.594459 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:10.594481 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:10.594577 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:10.594832 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 09:04:10.594972 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260620 09:04:10.598109 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260620 09:04:10.598388 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:10.598455 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bce866709141d89e83881b421fe69c, State: Initialized, Role: FOLLOWER
I20260620 09:04:10.598702 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:10.598964 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260620 09:04:10.599116 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:10.599184 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:10.599288 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:10.600268 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260620 09:04:10.600659 21497 leader_election.cc:302] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bce866709141d89e83881b421fe69c; no voters:
I20260620 09:04:10.600986 21497 leader_election.cc:288] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:10.601043 21507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:10.601329 21507 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [term 1 LEADER]: Becoming Leader. State: Replica: b5bce866709141d89e83881b421fe69c, State: Running, Role: LEADER
I20260620 09:04:10.601681 21507 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260620 09:04:10.602202 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:10.602825 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bce866709141d89e83881b421fe69c. Latest consensus state: current_term: 1 leader_uuid: "b5bce866709141d89e83881b421fe69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } } }
I20260620 09:04:10.603008 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:10.603238 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5bce866709141d89e83881b421fe69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bce866709141d89e83881b421fe69c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } } }
I20260620 09:04:10.603380 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:10.603509 21516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:10.605814 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:10.600789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34951 (local address 127.0.0.1:43371)
0620 09:04:10.600892 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:10.600898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:10.601118 (+ 220us) server_negotiation.cc:407] Connection header received
0620 09:04:10.601254 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:10.601257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:10.601299 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:10.601371 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:10.601759 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:10.602510 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:10.603643 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:10.603968 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:10.604072 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:10.605119 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:10.605122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:10.605125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:10.605138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:10.605163 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:10.605175 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:10.605250 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:10.605653 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 09:04:10.605686 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:04:10.606998 21516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:10.613693 21516 catalog_manager.cc:1347] Generated new cluster ID: 323742658d4b471c8db2549491dcec50
I20260620 09:04:10.613754 21516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:10.629343 21516 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:10.630227 21516 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:10.639955 21516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5bce866709141d89e83881b421fe69c: Generated new TSK 0
I20260620 09:04:10.640877 21516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:10.651386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21429
W20260620 09:04:10.656780 21414 connection.cc:535] client connection to 127.0.0.1:43371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43371): Connection reset by peer (error 104)
2026-06-20T09:04:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:10 dist-test-slave-scsf krb5kdc[21528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:10 dist-test-slave-scsf krb5kdc[21528](info): set up 2 sockets
Jun 20 09:04:10 dist-test-slave-scsf krb5kdc[21528](info): commencing operation
krb5kdc: starting...
W20260620 09:04:11.692844 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:11 dist-test-slave-scsf krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:11.823678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:11.935417 21541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:11.935729 21541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:11.935808 21541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:11.939963 21541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:11.940064 21541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:11.940104 21541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:11.940141 21541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:11.940177 21541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:11.945657 21541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:11.946753 21541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:11.947775 21541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:04:11.956336 21541 server_base.cc:1034] running on GCE node
W20260620 09:04:11.956250 21549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:11.956528 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:11.956272 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:11.957152 21541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:11.958262 21541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:11.959460 21541 hybrid_clock.cc:648] HybridClock initialized: now 1781946251959445 us; error 25 us; skew 500 ppm
Jun 20 09:04:11 dist-test-slave-scsf krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:11.962143 21541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:11.963166 21541 webserver.cc:460] Webserver started at http://127.0.0.1:46357/ using document root <none> and password file <none>
I20260620 09:04:11.963722 21541 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:11.963820 21541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:11.964048 21541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:11.966123 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "452f3b74221f4091a00515a4ea7f9e4b"
format_stamp: "Formatted at 2026-06-20 09:04:11 on dist-test-slave-scsf"
server_key: "54f195e5e8fdf3758a6219913662282a"
server_key_iv: "30a33509c4dc560a095ecfa4a25d685b"
server_key_version: "encryptionkey@0"
I20260620 09:04:11.966682 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "452f3b74221f4091a00515a4ea7f9e4b"
format_stamp: "Formatted at 2026-06-20 09:04:11 on dist-test-slave-scsf"
server_key: "54f195e5e8fdf3758a6219913662282a"
server_key_iv: "30a33509c4dc560a095ecfa4a25d685b"
server_key_version: "encryptionkey@0"
I20260620 09:04:11.970346 21541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 09:04:11.972641 21556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:11.973831 21541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:04:11.973973 21541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "452f3b74221f4091a00515a4ea7f9e4b"
format_stamp: "Formatted at 2026-06-20 09:04:11 on dist-test-slave-scsf"
server_key: "54f195e5e8fdf3758a6219913662282a"
server_key_iv: "30a33509c4dc560a095ecfa4a25d685b"
server_key_version: "encryptionkey@0"
I20260620 09:04:11.974086 21541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:11.978550 21541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:11.981339 21541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:11.981529 21541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:11.988435 21541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260620 09:04:11.988492 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260620 09:04:11.989437 21541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:11.992058 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:11.997620 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b: Bootstrap starting.
I20260620 09:04:11.999577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21541
I20260620 09:04:11.999768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:12.000054 392 external_mini_cluster.cc:1428] Setting key 7edbbfcfc2d7d95fa04833bb1c480200
I20260620 09:04:12.000077 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:12.000979 21609 log.cc:824] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:12.002867 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b: No bootstrap required, opened a new log
I20260620 09:04:12.004953 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:12.001167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:45133 (local address 127.0.0.1:36775)
0620 09:04:12.001652 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:12.001667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:12.001700 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:04:12.002314 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:12.002331 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:12.002591 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:12.002867 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:12.003680 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:12.003684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:12.003695 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:12.003704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:12.003772 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:12.003791 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:12.003887 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:12.004024 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 09:04:12.004194 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260620 09:04:12.006750 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 09:04:12.007009 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:12.007076 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452f3b74221f4091a00515a4ea7f9e4b, State: Initialized, Role: FOLLOWER
I20260620 09:04:12.007592 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:12.007535 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 09:04:12.007699 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:12.007766 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:12.007862 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:12.008783 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 09:04:12.009125 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452f3b74221f4091a00515a4ea7f9e4b; no voters:
I20260620 09:04:12.009430 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:12.009508 21618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:12.009727 21618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [term 1 LEADER]: Becoming Leader. State: Replica: 452f3b74221f4091a00515a4ea7f9e4b, State: Running, Role: LEADER
I20260620 09:04:12.010102 21618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 09:04:12.010821 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:12.011337 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452f3b74221f4091a00515a4ea7f9e4b. Latest consensus state: current_term: 1 leader_uuid: "452f3b74221f4091a00515a4ea7f9e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 09:04:12.011525 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:12.011741 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:12.009751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:56649 (local address 127.0.0.1:36775)
0620 09:04:12.009849 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:12.009853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:12.010237 (+ 384us) server_negotiation.cc:407] Connection header received
0620 09:04:12.010392 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:12.010395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:12.010428 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:12.010506 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:12.011272 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:12.011276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:12.011279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:12.011290 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:12.011316 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:12.011328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:12.011382 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:12.011555 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:04:12.011598 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:04:12.011771 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452f3b74221f4091a00515a4ea7f9e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452f3b74221f4091a00515a4ea7f9e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 09:04:12.011888 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:12.012166 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:12.015027 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:12.020082 21625 catalog_manager.cc:1347] Generated new cluster ID: b9f2b650b0224881b66067a5223feb2a
I20260620 09:04:12.020145 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:12.049337 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:12.049692 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:12.057893 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452f3b74221f4091a00515a4ea7f9e4b: Generated new TSK 0
I20260620 09:04:12.058751 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:12.068679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21541
2026-06-20T09:04:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:12 dist-test-slave-scsf krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:12 dist-test-slave-scsf krb5kdc[21640](info): set up 2 sockets
Jun 20 09:04:12 dist-test-slave-scsf krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260620 09:04:13.100852 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:13.219880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:13.327741 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:13.328105 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:13.328187 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:13.332459 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:13.332571 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:13.332619 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:13.332684 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:13.332726 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:13.338167 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:13.339278 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:13.340337 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:13.348802 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:13.348805 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:13.348822 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:13.349083 21653 server_base.cc:1034] running on GCE node
I20260620 09:04:13.349817 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:13.350893 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:13.352083 21653 hybrid_clock.cc:648] HybridClock initialized: now 1781946253352069 us; error 26 us; skew 500 ppm
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:13.354789 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:13.355644 21653 webserver.cc:460] Webserver started at http://127.0.0.1:35085/ using document root <none> and password file <none>
I20260620 09:04:13.356200 21653 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:13.356292 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:13.356573 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:13.358572 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "4beddd7ef3794a439c6c00c5b5d30c21"
format_stamp: "Formatted at 2026-06-20 09:04:13 on dist-test-slave-scsf"
server_key: "498fdd05ff236da17e7fadbb392ad9fa"
server_key_iv: "e43dd0eccc54b217bc4047cbd383867e"
server_key_version: "encryptionkey@0"
I20260620 09:04:13.359164 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "4beddd7ef3794a439c6c00c5b5d30c21"
format_stamp: "Formatted at 2026-06-20 09:04:13 on dist-test-slave-scsf"
server_key: "498fdd05ff236da17e7fadbb392ad9fa"
server_key_iv: "e43dd0eccc54b217bc4047cbd383867e"
server_key_version: "encryptionkey@0"
I20260620 09:04:13.362911 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:04:13.365411 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:13.366752 21653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:13.366912 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "4beddd7ef3794a439c6c00c5b5d30c21"
format_stamp: "Formatted at 2026-06-20 09:04:13 on dist-test-slave-scsf"
server_key: "498fdd05ff236da17e7fadbb392ad9fa"
server_key_iv: "e43dd0eccc54b217bc4047cbd383867e"
server_key_version: "encryptionkey@0"
I20260620 09:04:13.367041 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:13.392927 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:13.396270 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:13.396551 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:13.403882 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37957
I20260620 09:04:13.403903 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37957 every 8 connection(s)
I20260620 09:04:13.404928 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:13.405934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21653
I20260620 09:04:13.406088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:13.406359 392 external_mini_cluster.cc:1428] Setting key 63a5f72fd509478b545587911300f3d0
I20260620 09:04:13.407989 21721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:13.414600 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: Bootstrap starting.
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:04:13.417142 21721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:13.417990 21721 log.cc:824] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:13.420236 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: No bootstrap required, opened a new log
I20260620 09:04:13.421471 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:13.407779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:39205 (local address 127.0.0.1:37957)
0620 09:04:13.408170 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:13.408188 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:13.408236 (+ 48us) server_negotiation.cc:407] Connection header received
0620 09:04:13.408922 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:13.408938 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:13.409275 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:13.409543 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:13.410440 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:13.411548 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:13.412810 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:13.413298 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:13.413425 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:13.416541 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:13.416547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:13.416560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:13.416587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:13.418420 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:13.419106 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:13.419111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:13.419115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:13.419220 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:13.419543 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:13.419545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:13.419546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:13.419913 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:13.420063 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:13.420501 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 09:04:13.420763 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20260620 09:04:13.424062 21721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260620 09:04:13.424336 21721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:13.424378 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:13.424418 21721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4beddd7ef3794a439c6c00c5b5d30c21, State: Initialized, Role: FOLLOWER
I20260620 09:04:13.424860 21721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260620 09:04:13.424978 21721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:13.425060 21721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:13.425168 21721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:13.426152 21721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260620 09:04:13.426533 21721 leader_election.cc:302] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4beddd7ef3794a439c6c00c5b5d30c21; no voters:
I20260620 09:04:13.426910 21721 leader_election.cc:288] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:13.427003 21731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:13.427217 21731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [term 1 LEADER]: Becoming Leader. State: Replica: 4beddd7ef3794a439c6c00c5b5d30c21, State: Running, Role: LEADER
I20260620 09:04:13.427634 21731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260620 09:04:13.428217 21721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:13.429019 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4beddd7ef3794a439c6c00c5b5d30c21. Latest consensus state: current_term: 1 leader_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260620 09:04:13.429136 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:13.429435 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beddd7ef3794a439c6c00c5b5d30c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260620 09:04:13.429574 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21 [sys.catalog]: This master's current role is: LEADER
W20260620 09:04:13.433358 21746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:13.433471 21746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:13.433578 21740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:13.434567 21740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:13.435907 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:13.426456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:40507 (local address 127.0.0.1:37957)
0620 09:04:13.426585 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:13.426589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:13.426766 (+ 177us) server_negotiation.cc:407] Connection header received
0620 09:04:13.426908 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:13.426911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:13.426962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:13.427044 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:13.427970 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:13.428787 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:13.429919 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:13.430244 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:13.430977 (+ 733us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:13.431992 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:13.431994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:13.431997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:13.432009 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:13.433451 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:13.434059 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:13.434062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:13.434064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:13.434153 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:13.434534 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:13.434537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:13.434539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:13.434750 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:13.434898 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:13.435621 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 09:04:13.435715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:04:13.440277 21740 catalog_manager.cc:1347] Generated new cluster ID: 44fe84a7687340f0aedbcabfb8fffbca
I20260620 09:04:13.440372 21740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:13.461047 21740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:13.462357 21740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:13.479475 21740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4beddd7ef3794a439c6c00c5b5d30c21: Generated new TSK 0
I20260620 09:04:13.480384 21740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:13.484028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21653
W20260620 09:04:13.489435 21638 connection.cc:535] client connection to 127.0.0.1:37957 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37957): Connection reset by peer (error 104)
2026-06-20T09:04:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21752](info): set up 2 sockets
Jun 20 09:04:13 dist-test-slave-scsf krb5kdc[21752](info): commencing operation
krb5kdc: starting...
W20260620 09:04:14.536743 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:14 dist-test-slave-scsf krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:14.655663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:14.767470 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:14.767735 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:14.767812 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:14.771811 21765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:14.771873 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:14.771893 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:14.771909 21765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:14.771968 21765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:14.777122 21765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:14.778126 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:14.779158 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:14.787389 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:14.787379 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:14.787382 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:14.787623 21765 server_base.cc:1034] running on GCE node
I20260620 09:04:14.788229 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:14.789353 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:14.790551 21765 hybrid_clock.cc:648] HybridClock initialized: now 1781946254790525 us; error 43 us; skew 500 ppm
Jun 20 09:04:14 dist-test-slave-scsf krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:14.793243 21765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:14.794140 21765 webserver.cc:460] Webserver started at http://127.0.0.1:41597/ using document root <none> and password file <none>
I20260620 09:04:14.794694 21765 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:14.794786 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:14.795014 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:14.797084 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "165664892ab949dcb4bc70d68666e9d8"
format_stamp: "Formatted at 2026-06-20 09:04:14 on dist-test-slave-scsf"
server_key: "cf8791c87d6b814bb1f3882b6f63aa00"
server_key_iv: "06a3dfb966542286db578f5dc6c66c89"
server_key_version: "encryptionkey@0"
I20260620 09:04:14.797689 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "165664892ab949dcb4bc70d68666e9d8"
format_stamp: "Formatted at 2026-06-20 09:04:14 on dist-test-slave-scsf"
server_key: "cf8791c87d6b814bb1f3882b6f63aa00"
server_key_iv: "06a3dfb966542286db578f5dc6c66c89"
server_key_version: "encryptionkey@0"
I20260620 09:04:14.801396 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:04:14.803673 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:14.804913 21765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:14.805058 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "165664892ab949dcb4bc70d68666e9d8"
format_stamp: "Formatted at 2026-06-20 09:04:14 on dist-test-slave-scsf"
server_key: "cf8791c87d6b814bb1f3882b6f63aa00"
server_key_iv: "06a3dfb966542286db578f5dc6c66c89"
server_key_version: "encryptionkey@0"
I20260620 09:04:14.805163 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:14.826221 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:14.829725 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:14.829931 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:14.837054 21765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35587
I20260620 09:04:14.837080 21832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35587 every 8 connection(s)
I20260620 09:04:14.837997 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:14.840728 21833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:14.841033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21765
I20260620 09:04:14.841185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:14.841447 392 external_mini_cluster.cc:1428] Setting key e5adbbe25741ab619bd9a2014549802a
I20260620 09:04:14.847879 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8: Bootstrap starting.
I20260620 09:04:14.850685 21833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:14.851505 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:14.842855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:56933 (local address 127.0.0.1:35587)
0620 09:04:14.843275 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:14.843294 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:14.843337 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:04:14.844090 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:14.844113 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:14.844580 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:14.844969 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:14.845370 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:14.846884 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:14.847799 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:14.848480 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:14.848628 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:14.849504 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:14.849510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:14.849525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:14.850019 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:14.850118 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:14.850148 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:14.850279 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:14.850432 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 09:04:14.850569 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260620 09:04:14.851543 21833 log.cc:824] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:14.853760 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8: No bootstrap required, opened a new log
I20260620 09:04:14.854465 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:14.857553 21833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 09:04:14.857769 21833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:14.857837 21833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165664892ab949dcb4bc70d68666e9d8, State: Initialized, Role: FOLLOWER
I20260620 09:04:14.858322 21833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 09:04:14.858484 21833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:14.858552 21833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:14.858667 21833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:14.859774 21833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 09:04:14.860168 21833 leader_election.cc:302] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165664892ab949dcb4bc70d68666e9d8; no voters:
I20260620 09:04:14.860528 21833 leader_election.cc:288] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:14.860613 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:14.860841 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [term 1 LEADER]: Becoming Leader. State: Replica: 165664892ab949dcb4bc70d68666e9d8, State: Running, Role: LEADER
I20260620 09:04:14.861092 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:14.856532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34433 (local address 127.0.0.1:35587)
0620 09:04:14.856656 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:14.856663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:14.856865 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:04:14.857000 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:14.857003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:14.857049 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:14.857133 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:14.857397 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:14.858024 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:14.859188 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:14.859427 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:14.859504 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:14.860478 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:14.860484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:14.860487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:14.860503 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:14.860534 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:14.860548 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:14.860667 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:14.860932 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 09:04:14.860970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:04:14.861366 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 09:04:14.861882 21833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:14.863355 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165664892ab949dcb4bc70d68666e9d8. Latest consensus state: current_term: 1 leader_uuid: "165664892ab949dcb4bc70d68666e9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 09:04:14.863355 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165664892ab949dcb4bc70d68666e9d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165664892ab949dcb4bc70d68666e9d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 09:04:14.863509 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:14.863507 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:14.863914 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:14.867003 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:14.872251 21854 catalog_manager.cc:1347] Generated new cluster ID: b7af5e4a34dc4e269b35dbea37010b3b
I20260620 09:04:14.872323 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:14.885432 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:14.886739 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:14.897624 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165664892ab949dcb4bc70d68666e9d8: Generated new TSK 0
I20260620 09:04:14.898466 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:14.908200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21765
W20260620 09:04:14.913497 21750 connection.cc:535] client connection to 127.0.0.1:35587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35587): Connection reset by peer (error 104)
2026-06-20T09:04:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:14 dist-test-slave-scsf krb5kdc[21864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:14 dist-test-slave-scsf krb5kdc[21864](info): set up 2 sockets
Jun 20 09:04:14 dist-test-slave-scsf krb5kdc[21864](info): commencing operation
krb5kdc: starting...
W20260620 09:04:15.944696 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:16 dist-test-slave-scsf krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:04:16.061201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:16.166981 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:16.167301 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:16.167385 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:16.171316 21877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:16.171412 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:16.171451 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:16.171487 21877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:16.171519 21877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:16.176621 21877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:16.177641 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:16.178650 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:16.186825 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:16.186890 21877 server_base.cc:1034] running on GCE node
W20260620 09:04:16.186820 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:16.186820 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:16.187705 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:16.188768 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:16.189951 21877 hybrid_clock.cc:648] HybridClock initialized: now 1781946256189934 us; error 30 us; skew 500 ppm
Jun 20 09:04:16 dist-test-slave-scsf krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:16.197139 21877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:04:16.198048 21877 webserver.cc:460] Webserver started at http://127.0.0.1:36011/ using document root <none> and password file <none>
I20260620 09:04:16.198582 21877 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:16.198745 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:16.198978 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:16.201025 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "b085f72c8e9f4fc282b31433c3492de5"
format_stamp: "Formatted at 2026-06-20 09:04:16 on dist-test-slave-scsf"
server_key: "af54ac5cbb8c10fe789a800899a78d65"
server_key_iv: "5a82ea1a646483516c6c860bb0c1c2aa"
server_key_version: "encryptionkey@0"
I20260620 09:04:16.201593 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "b085f72c8e9f4fc282b31433c3492de5"
format_stamp: "Formatted at 2026-06-20 09:04:16 on dist-test-slave-scsf"
server_key: "af54ac5cbb8c10fe789a800899a78d65"
server_key_iv: "5a82ea1a646483516c6c860bb0c1c2aa"
server_key_version: "encryptionkey@0"
I20260620 09:04:16.205171 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:04:16.207779 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:16.209033 21877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:16.209185 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "b085f72c8e9f4fc282b31433c3492de5"
format_stamp: "Formatted at 2026-06-20 09:04:16 on dist-test-slave-scsf"
server_key: "af54ac5cbb8c10fe789a800899a78d65"
server_key_iv: "5a82ea1a646483516c6c860bb0c1c2aa"
server_key_version: "encryptionkey@0"
I20260620 09:04:16.209301 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:16.213815 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:16.216641 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:16.216843 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:16.223546 21877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37347
I20260620 09:04:16.223555 21944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37347 every 8 connection(s)
I20260620 09:04:16.224463 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:16.226260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21877
I20260620 09:04:16.226389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:16.226649 392 external_mini_cluster.cc:1428] Setting key 857e867691a63ad452b0aa22b38da74f
I20260620 09:04:16.227419 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:16.232144 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:16.227963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46379 (local address 127.0.0.1:37347)
0620 09:04:16.228351 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:16.228374 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:16.228415 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:04:16.229306 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:16.229328 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:16.229667 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:16.230042 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:16.230773 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:16.230778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:16.230793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:16.230804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:16.230866 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:16.230895 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:16.231034 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:16.231102 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:04:16.231258 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260620 09:04:16.234447 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5: Bootstrap starting.
I20260620 09:04:16.234948 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:16.236831 21945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:16.237694 21945 log.cc:824] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:16.238528 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:16.236924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:39421 (local address 127.0.0.1:37347)
0620 09:04:16.237033 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:16.237040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:16.237210 (+ 170us) server_negotiation.cc:407] Connection header received
0620 09:04:16.237304 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:16.237307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:16.237339 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:16.237414 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:16.238135 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:16.238139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:16.238141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:16.238154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:16.238181 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:16.238194 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:16.238288 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:16.238356 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:04:16.238396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:04:16.239773 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5: No bootstrap required, opened a new log
I20260620 09:04:16.243057 21945 raft_consensus.cc:348] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260620 09:04:16.243263 21945 raft_consensus.cc:374] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:16.243299 21945 raft_consensus.cc:729] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085f72c8e9f4fc282b31433c3492de5, State: Initialized, Role: FOLLOWER
I20260620 09:04:16.243628 21945 consensus_queue.cc:260] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260620 09:04:16.243767 21945 raft_consensus.cc:388] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:16.243829 21945 raft_consensus.cc:482] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:16.243932 21945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:16.244829 21945 raft_consensus.cc:504] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260620 09:04:16.245098 21945 leader_election.cc:302] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b085f72c8e9f4fc282b31433c3492de5; no voters:
I20260620 09:04:16.245337 21945 leader_election.cc:288] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:16.245451 21955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:16.245656 21955 raft_consensus.cc:686] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [term 1 LEADER]: Becoming Leader. State: Replica: b085f72c8e9f4fc282b31433c3492de5, State: Running, Role: LEADER
I20260620 09:04:16.245960 21955 consensus_queue.cc:237] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260620 09:04:16.246454 21945 sys_catalog.cc:564] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:16.247623 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b085f72c8e9f4fc282b31433c3492de5. Latest consensus state: current_term: 1 leader_uuid: "b085f72c8e9f4fc282b31433c3492de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260620 09:04:16.247629 21956 sys_catalog.cc:455] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b085f72c8e9f4fc282b31433c3492de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b085f72c8e9f4fc282b31433c3492de5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260620 09:04:16.247813 21956 sys_catalog.cc:458] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:16.247797 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:16.248631 21965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:16.251549 21965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:16.256659 21965 catalog_manager.cc:1347] Generated new cluster ID: 1df26d0997314d0a9ce838d8b59ea48d
I20260620 09:04:16.256732 21965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:16.270494 21965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:16.270795 21965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:16.276883 21965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b085f72c8e9f4fc282b31433c3492de5: Generated new TSK 0
I20260620 09:04:16.277515 21965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:16.285470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21877
2026-06-20T09:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:16 dist-test-slave-scsf krb5kdc[21976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:16 dist-test-slave-scsf krb5kdc[21976](info): set up 2 sockets
Jun 20 09:04:16 dist-test-slave-scsf krb5kdc[21976](info): commencing operation
krb5kdc: starting...
W20260620 09:04:17.316685 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:17.434715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:17.542328 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:17.542605 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:17.542652 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:17.546808 21989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:17.546876 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:17.546896 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:17.546913 21989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:17.546928 21989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:17.552500 21989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:17.553581 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:17.554659 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:17.562991 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:17.562991 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:17.563197 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:17.563093 21989 server_base.cc:1034] running on GCE node
I20260620 09:04:17.563820 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:17.564940 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:17.566141 21989 hybrid_clock.cc:648] HybridClock initialized: now 1781946257566127 us; error 27 us; skew 500 ppm
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:17.568782 21989 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:17.569607 21989 webserver.cc:460] Webserver started at http://127.0.98.62:37483/ using document root <none> and password file <none>
I20260620 09:04:17.570176 21989 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:17.570266 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:17.570495 21989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:17.572546 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "dd2701869cdd4cc8bf74f8663e77699e"
format_stamp: "Formatted at 2026-06-20 09:04:17 on dist-test-slave-scsf"
server_key: "60d58bf2d7e3fa819fa0a3a121da9c88"
server_key_iv: "d1dc844c85888ec030b9d0af32dac08e"
server_key_version: "encryptionkey@0"
I20260620 09:04:17.573120 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd2701869cdd4cc8bf74f8663e77699e"
format_stamp: "Formatted at 2026-06-20 09:04:17 on dist-test-slave-scsf"
server_key: "60d58bf2d7e3fa819fa0a3a121da9c88"
server_key_iv: "d1dc844c85888ec030b9d0af32dac08e"
server_key_version: "encryptionkey@0"
I20260620 09:04:17.576690 21989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:04:17.579002 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:17.580180 21989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:17.580317 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "dd2701869cdd4cc8bf74f8663e77699e"
format_stamp: "Formatted at 2026-06-20 09:04:17 on dist-test-slave-scsf"
server_key: "60d58bf2d7e3fa819fa0a3a121da9c88"
server_key_iv: "d1dc844c85888ec030b9d0af32dac08e"
server_key_version: "encryptionkey@0"
I20260620 09:04:17.580421 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:17.610667 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:17.613591 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:17.613837 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:17.620985 21989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45703
I20260620 09:04:17.621016 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45703 every 8 connection(s)
I20260620 09:04:17.621912 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:17.624571 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:17.629824 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e: Bootstrap starting.
I20260620 09:04:17.630800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 21989
I20260620 09:04:17.630919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:17.631182 392 external_mini_cluster.cc:1428] Setting key 4affa1d8fdc9d0abb58a898b0bf0b6a2
I20260620 09:04:17.632463 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:17.633250 22057 log.cc:824] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:17.635064 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e: No bootstrap required, opened a new log
I20260620 09:04:17.638510 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } }
I20260620 09:04:17.638767 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:17.638849 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2701869cdd4cc8bf74f8663e77699e, State: Initialized, Role: FOLLOWER
I20260620 09:04:17.639343 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } }
I20260620 09:04:17.639496 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:17.639570 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:17.639675 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 0 FOLLOWER]: Advancing to term 1
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:17.640803 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } }
I20260620 09:04:17.641155 22057 leader_election.cc:302] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2701869cdd4cc8bf74f8663e77699e; no voters:
I20260620 09:04:17.641458 22057 leader_election.cc:288] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:17.641544 22062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:17.641816 22062 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [term 1 LEADER]: Becoming Leader. State: Replica: dd2701869cdd4cc8bf74f8663e77699e, State: Running, Role: LEADER
I20260620 09:04:17.642145 22062 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } }
I20260620 09:04:17.642829 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:17.643306 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2701869cdd4cc8bf74f8663e77699e. Latest consensus state: current_term: 1 leader_uuid: "dd2701869cdd4cc8bf74f8663e77699e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } } }
I20260620 09:04:17.643498 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:17.643735 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2701869cdd4cc8bf74f8663e77699e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2701869cdd4cc8bf74f8663e77699e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45703 } } }
I20260620 09:04:17.643893 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:17.643956 22068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:17.646353 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:17.632269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54797 (local address 127.0.98.62:45703)
0620 09:04:17.632734 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:17.632749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:17.632784 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:04:17.633567 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:17.633585 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:17.633897 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:17.634219 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:17.635025 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:17.636035 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:17.636962 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:17.637440 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:17.637615 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:17.640511 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:17.640518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:17.640535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:17.640567 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:17.642517 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:17.643087 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:17.643093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:17.643098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:17.643242 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:17.643667 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:17.643674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:17.643678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:17.644216 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:17.645013 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:17.645332 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 09:04:17.645600 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260620 09:04:17.647092 22068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:17.649606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:17.653550 22068 catalog_manager.cc:1347] Generated new cluster ID: ea9adcd0c2d0436a8aac1c1e4addc4e9
I20260620 09:04:17.653602 22068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:17.660173 22068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:17.661371 22068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:17.664166 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:17.651764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39385 (local address 127.0.98.62:45703)
0620 09:04:17.651885 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:17.651889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:17.652118 (+ 229us) server_negotiation.cc:407] Connection header received
0620 09:04:17.652310 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:17.652313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:17.652354 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:17.652399 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:17.653319 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:17.657224 (+ 3905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:17.658376 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:17.658723 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:17.658837 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:17.660757 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:17.660761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:17.660764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:17.660787 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:17.662420 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:17.662942 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:17.662946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:17.662947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:17.663022 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:17.663359 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:17.663362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:17.663364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:17.663540 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:17.663660 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:17.663932 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 09:04:17.664019 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:04:17.683197 22068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2701869cdd4cc8bf74f8663e77699e: Generated new TSK 0
I20260620 09:04:17.683936 22068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:17.691335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 21989
W20260620 09:04:17.696765 21974 connection.cc:535] client connection to 127.0.98.62:45703 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45703): Connection reset by peer (error 104)
2026-06-20T09:04:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[22088](info): set up 2 sockets
Jun 20 09:04:17 dist-test-slave-scsf krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260620 09:04:18.720561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:18 dist-test-slave-scsf krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:18.840426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:18.949246 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:18.949504 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:18.949551 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:18.953511 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:18.953572 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:18.953590 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:18.953608 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:18.953622 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:18.958848 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:18.959867 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:18.960872 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:18.969033 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:18.969026 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:18.969033 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:18.969135 22101 server_base.cc:1034] running on GCE node
I20260620 09:04:18.969887 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:18.970968 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:18.972170 22101 hybrid_clock.cc:648] HybridClock initialized: now 1781946258972155 us; error 25 us; skew 500 ppm
Jun 20 09:04:18 dist-test-slave-scsf krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:18.974807 22101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:18.975587 22101 webserver.cc:460] Webserver started at http://127.0.98.62:43645/ using document root <none> and password file <none>
I20260620 09:04:18.976078 22101 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:18.976133 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:18.976300 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:18.978354 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e"
format_stamp: "Formatted at 2026-06-20 09:04:18 on dist-test-slave-scsf"
server_key: "e2d4be402b301c8630c471a11ab18d77"
server_key_iv: "1daaffc6ec25ffbb70ef07af930de196"
server_key_version: "encryptionkey@0"
I20260620 09:04:18.978883 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e"
format_stamp: "Formatted at 2026-06-20 09:04:18 on dist-test-slave-scsf"
server_key: "e2d4be402b301c8630c471a11ab18d77"
server_key_iv: "1daaffc6ec25ffbb70ef07af930de196"
server_key_version: "encryptionkey@0"
I20260620 09:04:18.982630 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:04:18.985040 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:18.986191 22101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:18.986322 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e"
format_stamp: "Formatted at 2026-06-20 09:04:18 on dist-test-slave-scsf"
server_key: "e2d4be402b301c8630c471a11ab18d77"
server_key_iv: "1daaffc6ec25ffbb70ef07af930de196"
server_key_version: "encryptionkey@0"
I20260620 09:04:18.986424 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:18.996049 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:18.998809 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:18.999009 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:19.005640 22101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42763
I20260620 09:04:19.005678 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42763 every 8 connection(s)
I20260620 09:04:19.006521 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:19.009183 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:19.014407 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: Bootstrap starting.
I20260620 09:04:19.016564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22101
I20260620 09:04:19.016695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:19.016898 22169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:19.016969 392 external_mini_cluster.cc:1428] Setting key c8fe946a011a36ac1aee5b8b309ba75d
I20260620 09:04:19.017767 22169 log.cc:824] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:19.019752 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: No bootstrap required, opened a new log
I20260620 09:04:19.023080 22169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260620 09:04:19.023305 22169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:19.023388 22169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9f7ab4cdb04f68b24af2bdbf82396e, State: Initialized, Role: FOLLOWER
I20260620 09:04:19.023871 22169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260620 09:04:19.024029 22169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:19.024101 22169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:19.024205 22169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:19.025300 22169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260620 09:04:19.025631 22169 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9f7ab4cdb04f68b24af2bdbf82396e; no voters:
I20260620 09:04:19.025821 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:19.018071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59811 (local address 127.0.98.62:42763)
0620 09:04:19.018568 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:19.018589 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:19.018633 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:04:19.019267 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:19.019283 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:19.019595 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:19.019937 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:19.020332 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:19.021351 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:19.022236 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:19.022697 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:19.022877 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:19.023811 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:19.023816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:19.023831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:19.024338 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:19.024404 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:19.024435 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:19.024592 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:19.024729 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 09:04:19.024856 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":174,"threads_started":1}
I20260620 09:04:19.025947 22169 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:19.026057 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:19.026254 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [term 1 LEADER]: Becoming Leader. State: Replica: 7e9f7ab4cdb04f68b24af2bdbf82396e, State: Running, Role: LEADER
I20260620 09:04:19.026584 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260620 09:04:19.027130 22169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:19.028174 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } } }
I20260620 09:04:19.028299 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:19.028723 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9f7ab4cdb04f68b24af2bdbf82396e. Latest consensus state: current_term: 1 leader_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9f7ab4cdb04f68b24af2bdbf82396e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } } }
I20260620 09:04:19.028862 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:19.029067 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 09:04:19.032275 22189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:19.032348 22189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:19.032411 22183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:19.033205 22183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:19.035902 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:19.031520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35539 (local address 127.0.98.62:42763)
0620 09:04:19.031617 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:19.031622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:19.031641 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:04:19.031780 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:19.031783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:19.031832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:19.031914 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:19.032206 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:19.032874 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:19.033964 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:19.034331 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:19.034449 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:19.035411 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:19.035415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:19.035418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:19.035432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:19.035460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:19.035473 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:19.035568 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:19.035731 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 09:04:19.035765 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:04:19.038635 22183 catalog_manager.cc:1347] Generated new cluster ID: 1b0899a384254675a3207b8378018467
I20260620 09:04:19.038708 22183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:19.062847 22183 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:19.063809 22183 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:19.071210 22183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9f7ab4cdb04f68b24af2bdbf82396e: Generated new TSK 0
I20260620 09:04:19.071905 22183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:19.077164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22101
W20260620 09:04:19.081923 22086 connection.cc:535] client connection to 127.0.98.62:42763 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42763): Connection reset by peer (error 104)
2026-06-20T09:04:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:19 dist-test-slave-scsf krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:19 dist-test-slave-scsf krb5kdc[22200](info): set up 2 sockets
Jun 20 09:04:19 dist-test-slave-scsf krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260620 09:04:20.104727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:20 dist-test-slave-scsf krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:20.223160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:20.329909 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:20.330235 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:20.330320 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:20.334301 22213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:20.334399 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:20.334439 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:20.334496 22213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:20.334535 22213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:20.339910 22213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:20.341018 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:20.342059 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:20.350548 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:20.350560 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:20.350570 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:20.350914 22213 server_base.cc:1034] running on GCE node
I20260620 09:04:20.351480 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:20.352531 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:20.353688 22213 hybrid_clock.cc:648] HybridClock initialized: now 1781946260353664 us; error 38 us; skew 500 ppm
Jun 20 09:04:20 dist-test-slave-scsf krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:20.356364 22213 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:20.357252 22213 webserver.cc:460] Webserver started at http://127.0.98.62:46145/ using document root <none> and password file <none>
I20260620 09:04:20.357817 22213 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:20.357908 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:20.358127 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:20.360116 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "2674739a8ec04024a6265a7176948800"
format_stamp: "Formatted at 2026-06-20 09:04:20 on dist-test-slave-scsf"
server_key: "42f55ef3dc8cd4be614fc517ebc42a1d"
server_key_iv: "d2e96e57919264f38fc32d05d9569675"
server_key_version: "encryptionkey@0"
I20260620 09:04:20.360710 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "2674739a8ec04024a6265a7176948800"
format_stamp: "Formatted at 2026-06-20 09:04:20 on dist-test-slave-scsf"
server_key: "42f55ef3dc8cd4be614fc517ebc42a1d"
server_key_iv: "d2e96e57919264f38fc32d05d9569675"
server_key_version: "encryptionkey@0"
I20260620 09:04:20.364228 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:04:20.366641 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:20.367887 22213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:20.368024 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "2674739a8ec04024a6265a7176948800"
format_stamp: "Formatted at 2026-06-20 09:04:20 on dist-test-slave-scsf"
server_key: "42f55ef3dc8cd4be614fc517ebc42a1d"
server_key_iv: "d2e96e57919264f38fc32d05d9569675"
server_key_version: "encryptionkey@0"
I20260620 09:04:20.368139 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:20.372663 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:20.375407 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:20.375598 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:20.382280 22213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45207
I20260620 09:04:20.382296 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45207 every 8 connection(s)
I20260620 09:04:20.383244 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:20.385998 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:20.388892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22213
I20260620 09:04:20.389037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:20.389331 392 external_mini_cluster.cc:1428] Setting key 68df74d9f6a6fe944b65ef3dc1ee0037
I20260620 09:04:20.392073 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800: Bootstrap starting.
I20260620 09:04:20.394451 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:20.390657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48757 (local address 127.0.98.62:45207)
0620 09:04:20.391130 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:20.391143 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:20.391181 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:04:20.391788 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:20.391804 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:20.392091 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:20.392418 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:20.393235 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:20.393240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:20.393250 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:20.393259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:20.393309 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:20.393328 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:20.393441 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:20.393563 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:04:20.393760 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":162,"threads_started":1}
I20260620 09:04:20.394726 22281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:20.395599 22281 log.cc:824] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:20.397537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:20.397938 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800: No bootstrap required, opened a new log
I20260620 09:04:20.401288 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:20.399587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50747 (local address 127.0.98.62:45207)
0620 09:04:20.399698 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:20.399703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:20.399879 (+ 176us) server_negotiation.cc:407] Connection header received
0620 09:04:20.400030 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:20.400033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:20.400070 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:20.400151 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:20.400896 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:20.400900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:20.400903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:20.400914 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:20.400938 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:20.400949 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:20.401042 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:20.401120 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:04:20.401156 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:04:20.401928 22281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260620 09:04:20.402220 22281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:20.402304 22281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674739a8ec04024a6265a7176948800, State: Initialized, Role: FOLLOWER
I20260620 09:04:20.402742 22281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260620 09:04:20.402884 22281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:20.402931 22281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:20.403000 22281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:20.403841 22281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260620 09:04:20.404126 22281 leader_election.cc:302] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2674739a8ec04024a6265a7176948800; no voters:
I20260620 09:04:20.404376 22281 leader_election.cc:288] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:20.404551 22291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:20.404781 22291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [term 1 LEADER]: Becoming Leader. State: Replica: 2674739a8ec04024a6265a7176948800, State: Running, Role: LEADER
I20260620 09:04:20.405143 22291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260620 09:04:20.405694 22281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:20.406716 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2674739a8ec04024a6265a7176948800. Latest consensus state: current_term: 1 leader_uuid: "2674739a8ec04024a6265a7176948800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } } }
I20260620 09:04:20.406908 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:20.406919 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2674739a8ec04024a6265a7176948800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2674739a8ec04024a6265a7176948800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } } }
I20260620 09:04:20.407058 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:20.407362 22300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:20.410485 22300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:20.415751 22300 catalog_manager.cc:1347] Generated new cluster ID: 92cea0d4064944a5975f626211b08ec8
I20260620 09:04:20.415829 22300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:20.437287 22300 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:20.437628 22300 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:20.444021 22300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2674739a8ec04024a6265a7176948800: Generated new TSK 0
I20260620 09:04:20.444749 22300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:20.447450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22213
2026-06-20T09:04:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:20 dist-test-slave-scsf krb5kdc[22312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:20 dist-test-slave-scsf krb5kdc[22312](info): set up 2 sockets
Jun 20 09:04:20 dist-test-slave-scsf krb5kdc[22312](info): commencing operation
krb5kdc: starting...
W20260620 09:04:21.481001 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:21.599737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:21.707070 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:21.707381 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:21.707453 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:21.711409 22325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:21.711506 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:21.711545 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:21.711606 22325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:21.711645 22325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:21.716850 22325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:21.717885 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:21.718928 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:21.727320 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:21.727344 22331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:21.727542 22325 server_base.cc:1034] running on GCE node
W20260620 09:04:21.727311 22333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:21.728189 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:21.729348 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:21.730502 22325 hybrid_clock.cc:648] HybridClock initialized: now 1781946261730481 us; error 32 us; skew 500 ppm
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:21.737010 22325 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:21.737891 22325 webserver.cc:460] Webserver started at http://127.0.98.62:42333/ using document root <none> and password file <none>
I20260620 09:04:21.738422 22325 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:21.738483 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:21.738646 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:21.740703 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "7251c5bc1f054da89ad9883710048ddc"
format_stamp: "Formatted at 2026-06-20 09:04:21 on dist-test-slave-scsf"
server_key: "c7932190ec66de8aeca655792819aff9"
server_key_iv: "e2f69f3ff26e31c747a3e0bb418d0c03"
server_key_version: "encryptionkey@0"
I20260620 09:04:21.741195 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "7251c5bc1f054da89ad9883710048ddc"
format_stamp: "Formatted at 2026-06-20 09:04:21 on dist-test-slave-scsf"
server_key: "c7932190ec66de8aeca655792819aff9"
server_key_iv: "e2f69f3ff26e31c747a3e0bb418d0c03"
server_key_version: "encryptionkey@0"
I20260620 09:04:21.745150 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:04:21.748313 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:21.749490 22325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:21.749634 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "7251c5bc1f054da89ad9883710048ddc"
format_stamp: "Formatted at 2026-06-20 09:04:21 on dist-test-slave-scsf"
server_key: "c7932190ec66de8aeca655792819aff9"
server_key_iv: "e2f69f3ff26e31c747a3e0bb418d0c03"
server_key_version: "encryptionkey@0"
I20260620 09:04:21.749749 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:21.761427 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:21.764166 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:21.764362 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:21.771435 22325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43099
I20260620 09:04:21.771458 22392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43099 every 8 connection(s)
I20260620 09:04:21.772360 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:21.775004 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:21.775537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22325
I20260620 09:04:21.775678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:21.775933 392 external_mini_cluster.cc:1428] Setting key edb90bbac64cf4a0c68c7f53023385d3
I20260620 09:04:21.781919 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc: Bootstrap starting.
I20260620 09:04:21.784574 22393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:21.785322 22393 log.cc:824] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:21.787607 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc: No bootstrap required, opened a new log
I20260620 09:04:21.791345 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:21.777375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41463 (local address 127.0.98.62:43099)
0620 09:04:21.777816 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:21.777836 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:21.777878 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:04:21.778651 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:21.778690 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:21.779162 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:21.779509 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:21.780221 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:21.781755 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:21.782683 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:21.783391 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:21.783547 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:21.786246 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:21.786252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:21.786267 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:21.786294 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:21.788146 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:21.788613 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:21.788618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:21.788624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:21.788756 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:21.789094 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:21.789100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:21.789103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:21.789606 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:21.789847 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:21.790084 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 09:04:21.790408 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
I20260620 09:04:21.792229 22393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } }
I20260620 09:04:21.792531 22393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:21.792600 22393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7251c5bc1f054da89ad9883710048ddc, State: Initialized, Role: FOLLOWER
I20260620 09:04:21.793079 22393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } }
I20260620 09:04:21.793244 22393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:21.793326 22393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:21.793419 22393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:21.794389 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:21.794526 22393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } }
I20260620 09:04:21.794868 22393 leader_election.cc:302] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7251c5bc1f054da89ad9883710048ddc; no voters:
I20260620 09:04:21.795163 22393 leader_election.cc:288] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:21.795291 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:21.795506 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [term 1 LEADER]: Becoming Leader. State: Replica: 7251c5bc1f054da89ad9883710048ddc, State: Running, Role: LEADER
I20260620 09:04:21.795921 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } }
I20260620 09:04:21.796408 22393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:21.797475 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7251c5bc1f054da89ad9883710048ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } } }
I20260620 09:04:21.797610 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:21.797858 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7251c5bc1f054da89ad9883710048ddc. Latest consensus state: current_term: 1 leader_uuid: "7251c5bc1f054da89ad9883710048ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7251c5bc1f054da89ad9883710048ddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43099 } } }
I20260620 09:04:21.797952 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:21.798027 22412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:21.801339 22412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:21.807263 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:21.797007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59407 (local address 127.0.98.62:43099)
0620 09:04:21.797109 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:21.797115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:21.798800 (+ 1685us) server_negotiation.cc:407] Connection header received
0620 09:04:21.798930 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:21.798934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:21.798980 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:21.799069 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:21.799923 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:21.800430 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:21.801307 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:21.801691 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:21.801789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:21.803720 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:21.803723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:21.803726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:21.803756 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:21.805446 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:21.805952 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:21.805955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:21.805957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:21.806039 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:21.806331 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:21.806334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:21.806336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:21.806535 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:21.806651 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:21.806976 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 09:04:21.807091 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:04:21.808955 22412 catalog_manager.cc:1347] Generated new cluster ID: 3003d7f58c8846f08bc48c76d5964ae1
I20260620 09:04:21.809019 22412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:21.827201 22412 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:21.828752 22412 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:21.835950 22412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7251c5bc1f054da89ad9883710048ddc: Generated new TSK 0
I20260620 09:04:21.836941 22412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:21.848577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22325
2026-06-20T09:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22424](info): set up 2 sockets
Jun 20 09:04:21 dist-test-slave-scsf krb5kdc[22424](info): commencing operation
krb5kdc: starting...
W20260620 09:04:22.892880 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:22 dist-test-slave-scsf krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:23.011122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:23.117748 22437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:23.118042 22437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:23.118113 22437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:23.122015 22437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:23.122108 22437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:23.122146 22437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:23.122203 22437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:23.122238 22437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:23.127328 22437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:23.128353 22437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:23.129352 22437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:23.137501 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:23.137574 22442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:23.137512 22445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:23.137800 22437 server_base.cc:1034] running on GCE node
I20260620 09:04:23.138343 22437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:23.139415 22437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:23.140595 22437 hybrid_clock.cc:648] HybridClock initialized: now 1781946263140586 us; error 31 us; skew 500 ppm
Jun 20 09:04:23 dist-test-slave-scsf krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:23.143205 22437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:23.144054 22437 webserver.cc:460] Webserver started at http://127.0.98.62:39731/ using document root <none> and password file <none>
I20260620 09:04:23.144663 22437 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:23.144752 22437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:23.144996 22437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:23.147125 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "6f09273fde68444a83e82a579d4ac727"
format_stamp: "Formatted at 2026-06-20 09:04:23 on dist-test-slave-scsf"
server_key: "7b40cb9214253ace2a9ca6dd23706d8d"
server_key_iv: "6048e3d3244b9c12c21aa8c703bfe5c0"
server_key_version: "encryptionkey@0"
I20260620 09:04:23.147686 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f09273fde68444a83e82a579d4ac727"
format_stamp: "Formatted at 2026-06-20 09:04:23 on dist-test-slave-scsf"
server_key: "7b40cb9214253ace2a9ca6dd23706d8d"
server_key_iv: "6048e3d3244b9c12c21aa8c703bfe5c0"
server_key_version: "encryptionkey@0"
I20260620 09:04:23.151328 22437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:04:23.153652 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:23.154810 22437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:04:23.154956 22437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "6f09273fde68444a83e82a579d4ac727"
format_stamp: "Formatted at 2026-06-20 09:04:23 on dist-test-slave-scsf"
server_key: "7b40cb9214253ace2a9ca6dd23706d8d"
server_key_iv: "6048e3d3244b9c12c21aa8c703bfe5c0"
server_key_version: "encryptionkey@0"
I20260620 09:04:23.155069 22437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:23.178792 22437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:23.181646 22437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:23.181902 22437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:23.188712 22437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39523
I20260620 09:04:23.188738 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39523 every 8 connection(s)
I20260620 09:04:23.189679 22437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:23.192303 22505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:23.197286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22437
I20260620 09:04:23.197391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:23.197664 392 external_mini_cluster.cc:1428] Setting key 516ae1b83e0f10e400b68cf7095a47a7
I20260620 09:04:23.197674 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727: Bootstrap starting.
I20260620 09:04:23.200313 22505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:23.201202 22505 log.cc:824] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:23.203243 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727: No bootstrap required, opened a new log
I20260620 09:04:23.207531 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:23.199002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52823 (local address 127.0.98.62:39523)
0620 09:04:23.199417 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:23.199436 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:23.199477 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:04:23.200289 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:23.200324 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:23.200779 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:23.201127 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:23.201556 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:23.202910 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:23.203849 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:23.204560 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:23.204730 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:23.205481 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:23.205488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:23.205504 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:23.206014 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:23.206090 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:23.206127 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:23.206237 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:23.206379 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:04:23.206580 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260620 09:04:23.207718 22505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260620 09:04:23.207978 22505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:23.208032 22505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f09273fde68444a83e82a579d4ac727, State: Initialized, Role: FOLLOWER
I20260620 09:04:23.208518 22505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260620 09:04:23.208683 22505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:23.208748 22505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:23.208849 22505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:23.209852 22505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260620 09:04:23.210168 22505 leader_election.cc:302] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f09273fde68444a83e82a579d4ac727; no voters:
I20260620 09:04:23.210420 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:23.210517 22505 leader_election.cc:288] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:23.210582 22510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:23.210780 22510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [term 1 LEADER]: Becoming Leader. State: Replica: 6f09273fde68444a83e82a579d4ac727, State: Running, Role: LEADER
I20260620 09:04:23.211113 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260620 09:04:23.211762 22505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:23.212764 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f09273fde68444a83e82a579d4ac727. Latest consensus state: current_term: 1 leader_uuid: "6f09273fde68444a83e82a579d4ac727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260620 09:04:23.212900 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:23.213223 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f09273fde68444a83e82a579d4ac727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f09273fde68444a83e82a579d4ac727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260620 09:04:23.213292 22524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:23.213318 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:23.216452 22524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:23.218451 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:23.212649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44353 (local address 127.0.98.62:39523)
0620 09:04:23.212795 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:23.212799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:23.213851 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 09:04:23.214050 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:23.214053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:23.214104 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:23.214161 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:23.214614 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:23.215353 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:23.216599 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:23.216933 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:23.217010 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:23.218011 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:23.218014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:23.218017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:23.218032 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:23.218059 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:23.218073 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:23.218155 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:23.218271 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 09:04:23.218325 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 09:04:23.222661 22524 catalog_manager.cc:1347] Generated new cluster ID: 7c34208667834ba0ac20e2e1d9b9b531
I20260620 09:04:23.222721 22524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:23.239753 22524 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:23.240796 22524 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:23.248553 22524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f09273fde68444a83e82a579d4ac727: Generated new TSK 0
I20260620 09:04:23.249310 22524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:23.253048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22437
2026-06-20T09:04:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:23 dist-test-slave-scsf krb5kdc[22536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:23 dist-test-slave-scsf krb5kdc[22536](info): set up 2 sockets
Jun 20 09:04:23 dist-test-slave-scsf krb5kdc[22536](info): commencing operation
krb5kdc: starting...
W20260620 09:04:24.292817 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:24 dist-test-slave-scsf krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:24.411940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:24.520612 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:24.520920 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:24.520990 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:24.525014 22549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:24.525096 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:24.525133 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:24.525187 22549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:24.525221 22549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:24.530483 22549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:24.531582 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:24.532645 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:04:24.541160 22549 server_base.cc:1034] running on GCE node
W20260620 09:04:24.541190 22554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:24.541160 22555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:24.541352 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:24.542006 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:24.543098 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:24.544282 22549 hybrid_clock.cc:648] HybridClock initialized: now 1781946264544272 us; error 25 us; skew 500 ppm
Jun 20 09:04:24 dist-test-slave-scsf krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:24.547685 22549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:24.548735 22549 webserver.cc:460] Webserver started at http://127.0.98.62:36641/ using document root <none> and password file <none>
I20260620 09:04:24.549333 22549 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:24.549397 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:24.549670 22549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:24.551808 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "5ba169f131434183a5a7bc062ae87148"
format_stamp: "Formatted at 2026-06-20 09:04:24 on dist-test-slave-scsf"
server_key: "aa16afbf3d84326cb90e953dfdb10c8c"
server_key_iv: "ba498b222370184c0c9ab6f114ba72a8"
server_key_version: "encryptionkey@0"
I20260620 09:04:24.552412 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ba169f131434183a5a7bc062ae87148"
format_stamp: "Formatted at 2026-06-20 09:04:24 on dist-test-slave-scsf"
server_key: "aa16afbf3d84326cb90e953dfdb10c8c"
server_key_iv: "ba498b222370184c0c9ab6f114ba72a8"
server_key_version: "encryptionkey@0"
I20260620 09:04:24.556156 22549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:04:24.558774 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:24.560009 22549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:24.560168 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "5ba169f131434183a5a7bc062ae87148"
format_stamp: "Formatted at 2026-06-20 09:04:24 on dist-test-slave-scsf"
server_key: "aa16afbf3d84326cb90e953dfdb10c8c"
server_key_iv: "ba498b222370184c0c9ab6f114ba72a8"
server_key_version: "encryptionkey@0"
I20260620 09:04:24.560300 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:24.565234 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:24.568264 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:24.568548 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:24.576161 22549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45717
I20260620 09:04:24.576188 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45717 every 8 connection(s)
I20260620 09:04:24.577167 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:24.577827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22549
I20260620 09:04:24.577981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:24.578260 392 external_mini_cluster.cc:1428] Setting key 803c859517ae18469324bf17d79b26a6
I20260620 09:04:24.580348 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:24.583958 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:24.579712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58463 (local address 127.0.98.62:45717)
0620 09:04:24.580165 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:24.580183 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:24.580225 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:04:24.581055 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:24.581075 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:24.581404 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:24.581785 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:24.582548 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:24.582556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:24.582571 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:24.582582 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:24.582642 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:24.582667 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:24.582817 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:24.582916 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:04:24.583061 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
I20260620 09:04:24.586752 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:24.587070 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148: Bootstrap starting.
I20260620 09:04:24.589516 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:24.590408 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:24.588771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46227 (local address 127.0.98.62:45717)
0620 09:04:24.588903 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:24.588907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:24.589085 (+ 178us) server_negotiation.cc:407] Connection header received
0620 09:04:24.589198 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:24.589200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:24.589226 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:24.589288 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:24.590009 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:24.590012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:24.590014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:04:24.590024 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:24.590047 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:04:24.590056 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:24.590124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:24.590230 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 09:04:24.590260 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:04:24.590430 22617 log.cc:824] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:24.592310 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148: No bootstrap required, opened a new log
I20260620 09:04:24.595850 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } }
I20260620 09:04:24.596081 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:04:24.596138 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba169f131434183a5a7bc062ae87148, State: Initialized, Role: FOLLOWER
I20260620 09:04:24.596614 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } }
I20260620 09:04:24.596800 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:04:24.596869 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:04:24.596966 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:24.597793 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } }
I20260620 09:04:24.598121 22617 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba169f131434183a5a7bc062ae87148; no voters:
I20260620 09:04:24.598416 22617 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:04:24.598527 22627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:24.598773 22627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba169f131434183a5a7bc062ae87148, State: Running, Role: LEADER
I20260620 09:04:24.599102 22627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } }
I20260620 09:04:24.599562 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:24.600735 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba169f131434183a5a7bc062ae87148. Latest consensus state: current_term: 1 leader_uuid: "5ba169f131434183a5a7bc062ae87148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } } }
I20260620 09:04:24.600764 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba169f131434183a5a7bc062ae87148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba169f131434183a5a7bc062ae87148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45717 } } }
I20260620 09:04:24.600935 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:24.601172 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:24.601871 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:24.604584 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:24.609738 22637 catalog_manager.cc:1347] Generated new cluster ID: bacf179638da480ea8703ee783b051e6
I20260620 09:04:24.609808 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:24.625312 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:24.625617 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:24.631258 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba169f131434183a5a7bc062ae87148: Generated new TSK 0
I20260620 09:04:24.631911 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:04:24.641310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22549
2026-06-20T09:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1387 ms)
[----------] 18 tests from AuthTokenIssuingTest (25050 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:24 dist-test-slave-scsf krb5kdc[22648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:24 dist-test-slave-scsf krb5kdc[22648](info): set up 2 sockets
Jun 20 09:04:24 dist-test-slave-scsf krb5kdc[22648](info): commencing operation
krb5kdc: starting...
W20260620 09:04:25.676826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:25 dist-test-slave-scsf krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:25.795605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:25.903816 22661 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:25.904107 22661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:25.904130 22661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:25.904166 22661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:25.908051 22661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:25.908110 22661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:25.908129 22661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:25.908145 22661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:25.908160 22661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:25.913235 22661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:25.914218 22661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:25.915169 22661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:25.923642 22666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:25.923645 22669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:25.923671 22667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:25.923952 22661 server_base.cc:1034] running on GCE node
I20260620 09:04:25.924609 22661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:25.925750 22661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:25.926923 22661 hybrid_clock.cc:648] HybridClock initialized: now 1781946265926909 us; error 24 us; skew 500 ppm
Jun 20 09:04:25 dist-test-slave-scsf krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:25.929741 22661 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:25.930617 22661 webserver.cc:460] Webserver started at http://127.0.98.62:43885/ using document root <none> and password file <none>
I20260620 09:04:25.931180 22661 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:25.931241 22661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:25.931457 22661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:25.933503 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "1db5eba2efa64b7c9c7e14a2c8737788"
format_stamp: "Formatted at 2026-06-20 09:04:25 on dist-test-slave-scsf"
server_key: "3b28580f4bbbcdb4a83cfcadc0e226b7"
server_key_iv: "99dae76a6c404deda99402abcfa76dd9"
server_key_version: "encryptionkey@0"
I20260620 09:04:25.934054 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "1db5eba2efa64b7c9c7e14a2c8737788"
format_stamp: "Formatted at 2026-06-20 09:04:25 on dist-test-slave-scsf"
server_key: "3b28580f4bbbcdb4a83cfcadc0e226b7"
server_key_iv: "99dae76a6c404deda99402abcfa76dd9"
server_key_version: "encryptionkey@0"
I20260620 09:04:25.938117 22661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:04:25.940683 22676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:25.941946 22661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:04:25.942077 22661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "1db5eba2efa64b7c9c7e14a2c8737788"
format_stamp: "Formatted at 2026-06-20 09:04:25 on dist-test-slave-scsf"
server_key: "3b28580f4bbbcdb4a83cfcadc0e226b7"
server_key_iv: "99dae76a6c404deda99402abcfa76dd9"
server_key_version: "encryptionkey@0"
I20260620 09:04:25.942185 22661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:25.970690 22661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:25.974174 22661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:25.974383 22661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:25.981104 22661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38621
I20260620 09:04:25.981405 22728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38621 every 8 connection(s)
I20260620 09:04:25.982080 22661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:25.984784 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:25.988324 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:25.991761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22661
I20260620 09:04:25.991909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:25.992158 392 external_mini_cluster.cc:1428] Setting key 110272256191e79e8216d687eac80c9d
Jun 20 09:04:25 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 09:04:26.003544 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:25.990614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.98.62:38621)
0620 09:04:25.991114 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:25.991131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:25.991322 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:04:25.992430 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:25.992434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:25.992654 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:25.992862 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:25.993725 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:25.994574 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:25.995540 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:25.995900 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:25.996062 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:25.998779 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:25.998785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:25.998794 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:25.998815 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.000524 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.000994 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.000999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.001000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.001102 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.001469 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.001471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.001473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.001851 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.001952 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.002485 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.002736 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":35712,"thread_start_us":151,"threads_started":1}
I20260620 09:04:26.003551 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:25.990614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38621 (local address 127.0.0.1:35594)
0620 09:04:25.991163 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:25.991224 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:25.992134 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:25.992857 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:25.992862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:25.993326 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:25.993560 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:25.993572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:25.994702 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:25.994706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:25.995424 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:25.995432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:25.996050 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:25.996054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:25.996158 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:25.996835 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:25.996851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:25.998576 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.000644 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.000652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.000660 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.000891 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.001184 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.001186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.001187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.001355 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.002022 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.002031 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.002180 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.002383 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.002837 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":97,"thread_start_us":131,"threads_started":1}
I20260620 09:04:26.006206 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:26.007145 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.006781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45930)
0620 09:04:26.006947 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.007043 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 09:04:26.007397 22678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
I20260620 09:04:26.008258 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.007997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45942)
0620 09:04:26.008123 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.008182 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:04:26.008431 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34889: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:04:26.020107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34889
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:04:26.058635 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } attempt: 1
I20260620 09:04:26.059667 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.059275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45952)
0620 09:04:26.059470 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.059554 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 09:04:26.060621 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.060331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45968)
0620 09:04:26.060474 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.060538 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 09:04:26.060839 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34889: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
I20260620 09:04:26.128993 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } attempt: 2
I20260620 09:04:26.129998 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.129595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45982)
0620 09:04:26.129776 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.129858 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 09:04:26.131927 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:45990)
0620 09:04:26.131748 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.131817 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 09:04:26.132123 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34889: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34889: connect: Connection refused (error 111)
W20260620 09:04:26.134704 22736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:26.135020 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:26.135092 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:26.135177 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:26.139227 22736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:26.139315 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:26.139362 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:26.139397 22736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:26.139429 22736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:26.144869 22736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:26.146039 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:26.147111 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:26.155783 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:26.155763 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:26.155754 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:26.156102 22736 server_base.cc:1034] running on GCE node
I20260620 09:04:26.156752 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:26.157895 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:26.159088 22736 hybrid_clock.cc:648] HybridClock initialized: now 1781946266159068 us; error 32 us; skew 500 ppm
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:26.162108 22736 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:04:26.162971 22736 webserver.cc:460] Webserver started at http://127.0.98.61:40409/ using document root <none> and password file <none>
I20260620 09:04:26.163533 22736 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:26.163625 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:26.163864 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:26.165946 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data/instance:
uuid: "833d2c0e3be643f8841b1aa40600e564"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "38641958e4b212c047ed5fc7b469e21f"
server_key_iv: "5f9a539519837077fee122638880dfa5"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.166508 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal/instance:
uuid: "833d2c0e3be643f8841b1aa40600e564"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "38641958e4b212c047ed5fc7b469e21f"
server_key_iv: "5f9a539519837077fee122638880dfa5"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.170377 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:04:26.172941 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:26.174270 22736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:04:26.174408 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "833d2c0e3be643f8841b1aa40600e564"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "38641958e4b212c047ed5fc7b469e21f"
server_key_iv: "5f9a539519837077fee122638880dfa5"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.174516 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:26.195089 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:26.198173 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:26.198385 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:26.205160 22736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34889
I20260620 09:04:26.205451 22803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34889 every 8 connection(s)
I20260620 09:04:26.206147 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:04:26.208705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22736
I20260620 09:04:26.208828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-1/wal/instance
I20260620 09:04:26.209117 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:26.209143 392 external_mini_cluster.cc:1428] Setting key 124e3372ce9838ea6dc775ed9e43c835
I20260620 09:04:26.213248 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:26.223651 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } attempt: 3
I20260620 09:04:26.227388 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.215176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.98.62:38621)
0620 09:04:26.215342 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.215346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.216337 (+ 991us) server_negotiation.cc:407] Connection header received
0620 09:04:26.217376 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.217379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.217437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.217543 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.218646 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.219182 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.220289 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.220708 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.220825 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.223347 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.223350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.223352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.223376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.225301 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.225871 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.225877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.225880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.225981 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.226317 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.226322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.226323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.226504 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.226608 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.227093 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.227214 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:04:26.228318 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.215689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38621 (local address 127.0.0.1:35598)
0620 09:04:26.216189 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.216229 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.217170 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.217708 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.217716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.218031 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.218502 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.218514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.219315 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.219321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.220161 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.220170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.220882 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.220886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.221030 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.221645 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.221661 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.223152 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.225426 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.225432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.225447 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.225770 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.226083 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.226086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.226088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.226221 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.226606 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.226613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.226855 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.227041 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.227348 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":143,"threads_started":1}
I20260620 09:04:26.229877 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:04:26.234972 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.224117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:46000)
0620 09:04:26.224343 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.224358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.224465 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.225186 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.225189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.225412 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.225544 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.225551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.227200 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.227206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.228136 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.228150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.228881 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.228884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.229055 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.229421 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.229435 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.231098 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.233300 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.233302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.233303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.233541 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.233838 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.233841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.233843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.233958 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.234445 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.234450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.234546 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.234670 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.234809 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 09:04:26.235145 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.224486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.98.61:34889)
0620 09:04:26.224843 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.224852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.224872 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:04:26.224906 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.224909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.225061 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.225203 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.225783 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.227039 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.228251 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.228758 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.228882 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.231261 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.231270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.231285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.231314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.233192 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.233675 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.233679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.233681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.233752 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.234041 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.234044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.234045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.234332 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.234521 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.234790 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.234922 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":169,"threads_started":1}
I20260620 09:04:26.235599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43191
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:04:26.237259 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:26.237999 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.237720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39456)
0620 09:04:26.237903 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.237947 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 09:04:26.238888 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.238451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39472)
0620 09:04:26.238765 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.238816 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
W20260620 09:04:26.239061 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
I20260620 09:04:26.240723 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.230384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:46004)
0620 09:04:26.230880 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.230894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.230975 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.231249 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.231252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.231447 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.231606 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.231612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.232660 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.232664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.233497 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.233504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.234042 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.234045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.234158 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.234990 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.235006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.237178 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.239206 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.239214 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.239215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.239389 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.239720 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.239724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.239726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.239838 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.240295 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.240297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.240367 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.240510 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.240621 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425}
I20260620 09:04:26.240975 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.230462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.61:34889)
0620 09:04:26.230809 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.230819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.231054 (+ 235us) server_negotiation.cc:407] Connection header received
0620 09:04:26.231081 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.231083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.231128 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.231211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.231746 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.232553 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.233601 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.233924 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.234066 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.237327 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.237332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.237336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.237364 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.239109 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.239496 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.239502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.239507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.239599 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.239933 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.239940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.239943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.240190 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.240298 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.240570 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.240709 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260620 09:04:26.241639 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:26.242353 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.242031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39480)
0620 09:04:26.242163 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.242268 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:04:26.242578 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
I20260620 09:04:26.243474 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.243199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39492)
0620 09:04:26.243337 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.243399 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 09:04:26.243624 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
I20260620 09:04:26.270211 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 1
I20260620 09:04:26.271104 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.270813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39494)
0620 09:04:26.270968 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.271029 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 09:04:26.271842 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.271614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39498)
0620 09:04:26.271753 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.271794 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:04:26.271965 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
I20260620 09:04:26.292922 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 1
I20260620 09:04:26.293812 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.293480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39512)
0620 09:04:26.293643 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.293713 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 09:04:26.294554 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.294345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39518)
0620 09:04:26.294449 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.294499 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:04:26.294708 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
I20260620 09:04:26.347122 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 2
I20260620 09:04:26.347954 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.347632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39530)
0620 09:04:26.347770 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.347838 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 09:04:26.348750 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.348547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39534)
0620 09:04:26.348645 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.348692 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:04:26.348927 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
W20260620 09:04:26.349191 22811 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:26.349459 22811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:26.349521 22811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:26.349627 22811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:26.353708 22811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:26.353766 22811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:26.353786 22811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:26.353826 22811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:26.353859 22811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:26.359123 22811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38621,127.0.98.61:34889,127.0.98.60:43191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:26.360160 22811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:26.361233 22811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:04:26.362895 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 2
I20260620 09:04:26.363722 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.363385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39540)
0620 09:04:26.363545 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.363622 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 09:04:26.364530 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.364282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39550)
0620 09:04:26.364396 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.364465 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:04:26.364661 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43191: connect: Connection refused (error 111)
W20260620 09:04:26.370122 22818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:26.370074 22819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:26.370276 22811 server_base.cc:1034] running on GCE node
W20260620 09:04:26.370079 22821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:26.371023 22811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:26.372085 22811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:26.373263 22811 hybrid_clock.cc:648] HybridClock initialized: now 1781946266373232 us; error 45 us; skew 500 ppm
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:26.375991 22811 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:04:26.376900 22811 webserver.cc:460] Webserver started at http://127.0.98.60:37781/ using document root <none> and password file <none>
I20260620 09:04:26.377470 22811 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:26.377532 22811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:26.377705 22811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:26.379961 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data/instance:
uuid: "43a9ea312cca4d04ad64116f131ba19f"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "b45b36bed0bfdbb2655437ea02ee046a"
server_key_iv: "e9b0afcb3e42751dbe4a61f3a97f0413"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.380574 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal/instance:
uuid: "43a9ea312cca4d04ad64116f131ba19f"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "b45b36bed0bfdbb2655437ea02ee046a"
server_key_iv: "e9b0afcb3e42751dbe4a61f3a97f0413"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.384559 22811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:04:26.386963 22828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:26.388175 22811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:26.388314 22811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "43a9ea312cca4d04ad64116f131ba19f"
format_stamp: "Formatted at 2026-06-20 09:04:26 on dist-test-slave-scsf"
server_key: "b45b36bed0bfdbb2655437ea02ee046a"
server_key_iv: "e9b0afcb3e42751dbe4a61f3a97f0413"
server_key_version: "encryptionkey@0"
I20260620 09:04:26.388427 22811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:26.403075 22811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:26.406339 22811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:26.406548 22811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:26.413162 22811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43191
I20260620 09:04:26.413443 22880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43191 every 8 connection(s)
I20260620 09:04:26.414141 22811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:04:26.416839 22881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:26.420485 22881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:26.422680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 22811
I20260620 09:04:26.422833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781945987698834-392-0/minicluster-data/master-2/wal/instance
I20260620 09:04:26.423164 392 external_mini_cluster.cc:1428] Setting key 9e711c94fa95f1984f7e1dc028c42e40
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:04:26.434574 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.422278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.0.98.62:38621)
0620 09:04:26.422419 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.422423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.423351 (+ 928us) server_negotiation.cc:407] Connection header received
0620 09:04:26.424549 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.424556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.424604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.424669 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.425700 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.426209 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.427273 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.427550 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.427627 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.430410 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.430414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.430416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.430440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.432140 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.432805 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.432811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.432816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.432929 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.433345 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.433352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.433357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.433582 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.433693 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.434235 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.434389 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:04:26.435391 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.422576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38621 (local address 127.0.0.1:35602)
0620 09:04:26.423165 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.423222 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.424269 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.424744 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.424755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.425188 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.425548 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.425560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.426325 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.426328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.427150 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.427159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.427680 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.427684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.427853 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.428498 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.428515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.430200 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.432252 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.432259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.432266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.432696 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.433030 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.433033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.433035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.433210 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.433709 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.433714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.433951 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.434143 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.434490 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":35,"spinlock_wait_cycles":2944,"thread_start_us":198,"threads_started":1}
I20260620 09:04:26.435391 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.423841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.98.60:43191)
0620 09:04:26.424195 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.424206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.424246 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:04:26.424393 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.424414 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.424698 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.424880 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.425543 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.426532 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.427370 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.427808 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.427904 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.430878 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.430884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.430894 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.430935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.432774 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.433296 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.433304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.433307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.433399 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.433665 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.433669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.433670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.433957 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.434144 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.434378 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.434546 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":1280,"thread_start_us":83,"threads_started":1}
I20260620 09:04:26.437191 22881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:26.445659 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.437647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34889 (local address 127.0.0.1:46018)
0620 09:04:26.437754 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.437771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.437854 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.438235 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.438238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.438400 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.438487 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.438492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.439234 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.439239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.439738 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.439746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.440274 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.440278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.440375 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.440849 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.440859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.442042 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.443955 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.443959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.443961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.444190 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.444487 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.444490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.444492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.444583 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.445147 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.445150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.445250 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.445371 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.445496 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:04:26.445668 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.437907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.98.61:34889)
0620 09:04:26.438042 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.438046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.438057 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:04:26.438094 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.438096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.438135 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.438223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.438601 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.439127 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.439859 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.440178 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.440280 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.442176 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.442179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.442180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.442201 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.443841 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.444293 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.444295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.444296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.444364 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.444676 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.444680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.444682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.444954 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.445052 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.445440 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.445558 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:04:26.446511 22881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:26.446642 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.437711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.0.98.62:38621)
0620 09:04:26.437823 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.437827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.438066 (+ 239us) server_negotiation.cc:407] Connection header received
0620 09:04:26.438175 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.438180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.438227 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.438314 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.438896 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.439371 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.440121 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.440333 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.440438 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.442768 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.442772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.442774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.442798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.444549 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.444996 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.445117 (+ 121us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.445121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.445205 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.445499 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.445502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.445504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.445701 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.446006 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.446366 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.446495 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:04:26.448067 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 3
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:04:26.455804 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } attempt: 3
I20260620 09:04:26.456432 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.447029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39568)
0620 09:04:26.447171 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.447192 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.447253 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.447613 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.447616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.447782 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.447894 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.447901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.449010 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.449016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.449785 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.449796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.450366 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.450370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.450489 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.450948 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.450962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.452222 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.454791 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.454794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.454798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.455040 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.455325 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.455330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.455333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.455427 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.455858 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.455861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.455960 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.456067 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.456230 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":21}
I20260620 09:04:26.457479 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.447092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.98.60:43191)
0620 09:04:26.447397 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.447403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.447417 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:04:26.447448 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.447450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.447494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.447569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.448015 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.448883 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.449895 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.450248 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.450349 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.452367 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.452370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.452373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.452394 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.454672 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.455150 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.455156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.455158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.455225 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.455516 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.455522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.455524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.455745 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.457120 (+ 1375us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.457200 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.457318 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260620 09:04:26.457686 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.447958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.98.61:34889)
0620 09:04:26.448049 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.448054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.448070 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:04:26.448197 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.448200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.448241 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.448323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.449024 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.449621 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.450637 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.450866 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.450970 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.453872 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.453876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.453879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.453903 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.455656 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.456157 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.456161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.456163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.456257 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.456643 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.456647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.456648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.456858 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.457041 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.457432 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.457524 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:04:26.459074 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.448418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39580)
0620 09:04:26.448546 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.448560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.448651 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.449256 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.449262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.449451 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.449554 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.449560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.451352 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.451358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.452410 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.452422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.453122 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.453125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.453231 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.453682 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.453697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.455016 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.457365 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.457369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.457371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.457628 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.457960 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.457963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.457965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.458058 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.458536 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.458539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.458652 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.458758 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.458902 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 09:04:26.459583 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.448649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.0.98.60:43191)
0620 09:04:26.449033 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.449042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.449053 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:04:26.449082 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.449084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.449121 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.449206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.449702 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.451177 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.452614 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.453021 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.453116 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.455156 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.455160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.455163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.455186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.457247 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.457744 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.457748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.457752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.457858 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.458170 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.458172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.458174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.458391 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.459250 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.459326 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.459427 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":307,"thread_start_us":181,"threads_started":1}
Jun 20 09:04:26 dist-test-slave-scsf krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:04:26.461419 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:26.461392 22881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: Bootstrap starting.
I20260620 09:04:26.462796 22703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1db5eba2efa64b7c9c7e14a2c8737788"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35612
I20260620 09:04:26.464169 22881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:26.465073 22881 log.cc:824] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:26.465288 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.456167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43191 (local address 127.0.0.1:39588)
0620 09:04:26.456441 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:26.456480 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:26.456559 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:26.457228 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:26.457231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:26.457412 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:26.457503 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.457510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.458570 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.458573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.459309 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:26.459320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.459905 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.459908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:26.459989 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.460372 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:26.460383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:26.461740 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:26.463689 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.463692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.463694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.463944 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.464256 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:26.464259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:26.464261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:26.464353 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:26.464801 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:26.464804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:26.464903 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 09:04:26.465020 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:04:26.465148 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 09:04:26.465378 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.456708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.0.98.60:43191)
0620 09:04:26.456998 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.457005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.457017 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:04:26.457048 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.457053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.457101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.457606 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.457709 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.458441 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.459450 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.459811 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.459904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.461872 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.461875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.461878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.461900 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.463569 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.464057 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.464060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.464062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.464148 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.464479 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.464483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.464484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.464689 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.464791 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.465082 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.465196 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
I20260620 09:04:26.467252 22881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: No bootstrap required, opened a new log
I20260620 09:04:26.468240 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: Bootstrap starting.
I20260620 09:04:26.468240 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: Bootstrap starting.
I20260620 09:04:26.470835 22729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:26.470857 22804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:26.471285 22881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.471526 22881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a9ea312cca4d04ad64116f131ba19f, State: Initialized, Role: FOLLOWER
I20260620 09:04:26.471685 22804 log.cc:824] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:26.471685 22729 log.cc:824] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:26.471875 22881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.472749 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.473044 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.473364 22881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:26.473750 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: No bootstrap required, opened a new log
I20260620 09:04:26.473750 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: No bootstrap required, opened a new log
I20260620 09:04:26.477967 22804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.477917 22729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.478155 22729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db5eba2efa64b7c9c7e14a2c8737788, State: Initialized, Role: FOLLOWER
I20260620 09:04:26.478155 22804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833d2c0e3be643f8841b1aa40600e564, State: Initialized, Role: FOLLOWER
I20260620 09:04:26.478500 22729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.478525 22804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.479148 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.479342 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.479861 22729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:26.479861 22804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:26.479912 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:04:26.480023 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:26.477229 22811 master_runner.cc:186] Error getting master registration for 127.0.98.62:38621: OK, instance_id { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" instance_seqno: 1781946265979325 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:04:26.480770 22811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:04:26.481369 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.482331 22703 leader_election.cc:288] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [CANDIDATE]: Term 1 election: Requested vote from peers 833d2c0e3be643f8841b1aa40600e564 (127.0.98.61:34889), 43a9ea312cca4d04ad64116f131ba19f (127.0.98.60:43191)
W20260620 09:04:26.483052 22902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 09:04:26.483031 22855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43a9ea312cca4d04ad64116f131ba19f"
W20260620 09:04:26.483167 22902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:26.483282 22855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:26.484725 22855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db5eba2efa64b7c9c7e14a2c8737788 in term 1.
I20260620 09:04:26.485174 22778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "833d2c0e3be643f8841b1aa40600e564"
I20260620 09:04:26.485414 22778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:26.486754 22778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db5eba2efa64b7c9c7e14a2c8737788 in term 1.
I20260620 09:04:26.487432 22678 leader_election.cc:302] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1db5eba2efa64b7c9c7e14a2c8737788, 43a9ea312cca4d04ad64116f131ba19f, 833d2c0e3be643f8841b1aa40600e564; no voters:
I20260620 09:04:26.487703 22903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:26.487911 22903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [term 1 LEADER]: Becoming Leader. State: Replica: 1db5eba2efa64b7c9c7e14a2c8737788, State: Running, Role: LEADER
I20260620 09:04:26.488258 22903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } }
I20260620 09:04:26.490114 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db5eba2efa64b7c9c7e14a2c8737788. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.490231 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:26.491505 22736 master_runner.cc:263] No leader master found from master 833d2c0e3be643f8841b1aa40600e564
I20260620 09:04:26.491564 22736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 09:04:26.492910 22927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:26.492978 22927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:26.493189 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.493861 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.496238 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.484678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.0.98.61:34889)
0620 09:04:26.484805 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.484809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.485239 (+ 430us) server_negotiation.cc:407] Connection header received
0620 09:04:26.485505 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.485508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.485560 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.485618 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.486695 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.487493 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.489049 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.489389 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.489463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.491417 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.491424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.491427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.491452 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.493609 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.494755 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.494759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.494761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.494846 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.495172 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.495176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.495178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.495408 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.495547 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.495940 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.496070 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 09:04:26.496595 22925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:26.496675 22925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:26.497466 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.484631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.0.98.62:38621)
0620 09:04:26.484756 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.484763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.485767 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 09:04:26.485932 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.485935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.485981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.486034 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.486914 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.487625 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.489049 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.489389 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.489463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.493386 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.493389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.493392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.493416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.495089 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.496080 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.496084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.496085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.496162 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.496521 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.496524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.496526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.496749 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.496847 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.497204 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.497331 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:04:26.498809 22937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:26.499807 22937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:26.506011 22855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [term 1 FOLLOWER]: Refusing update from remote peer 1db5eba2efa64b7c9c7e14a2c8737788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:04:26.506740 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:04:26.507231 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:26.487058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.0.98.60:43191)
0620 09:04:26.487167 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.487171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.498225 (+ 11054us) server_negotiation.cc:407] Connection header received
0620 09:04:26.498400 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.498403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.498454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.498511 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.499480 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.500257 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.501373 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:26.501699 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:26.501803 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:26.503470 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:26.503474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:26.503477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:26.503505 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:26.505210 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.505698 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.505702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.505704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.505799 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:26.506211 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:26.506214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:26.506216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:26.506440 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:26.506542 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:26.506956 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 09:04:26.507078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:04:26.508428 22778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [term 1 FOLLOWER]: Refusing update from remote peer 1db5eba2efa64b7c9c7e14a2c8737788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:04:26.509301 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:04:26.513978 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db5eba2efa64b7c9c7e14a2c8737788. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.514111 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.515048 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db5eba2efa64b7c9c7e14a2c8737788. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.515187 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.518308 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.518427 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:26.519599 22937 catalog_manager.cc:1347] Generated new cluster ID: 6d96fb546b4049cf81dd7ebe3314bfdb
I20260620 09:04:26.519659 22937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:26.523438 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.523546 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788 [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:26.524631 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.524748 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.525041 22941 mvcc.cc:205] Tried to move back new op lower bound from 7298851907598299136 to 7298851907539431424. Current Snapshot: MvccSnapshot[applied={T|T < 7298851907598299136}]
I20260620 09:04:26.525143 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db5eba2efa64b7c9c7e14a2c8737788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38621 } } peers { permanent_uuid: "833d2c0e3be643f8841b1aa40600e564" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34889 } } peers { permanent_uuid: "43a9ea312cca4d04ad64116f131ba19f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43191 } } }
I20260620 09:04:26.525259 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:26.532691 22937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:26.533973 22937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:26.556293 22937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db5eba2efa64b7c9c7e14a2c8737788: Generated new TSK 0
I20260620 09:04:26.557345 22937 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:04:26.596280 22887 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.593111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.0.98.60:43191)
0620 09:04:26.593246 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.593251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.595198 (+ 1947us) server_negotiation.cc:407] Connection header received
0620 09:04:26.595409 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.595412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.595461 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.595512 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.596125 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39610: BlockingRecv error: recv got EOF from 127.0.0.1:39610 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:04:26.597895 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.593923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.0.98.62:38621)
0620 09:04:26.594043 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.594048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.597022 (+ 2974us) server_negotiation.cc:407] Connection header received
0620 09:04:26.597185 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.597188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.597247 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.597297 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.597774 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35636: BlockingRecv error: recv got EOF from 127.0.0.1:35636 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 09:04:26.599386 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.594618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.98.61:34889)
0620 09:04:26.594736 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.594741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.598483 (+ 3742us) server_negotiation.cc:407] Connection header received
0620 09:04:26.598661 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.598664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.598730 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.598781 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.599272 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46044: BlockingRecv error: recv got EOF from 127.0.0.1:46044 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 09:04:26.604091 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.602464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.0.98.61:34889)
0620 09:04:26.602576 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.602580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.603130 (+ 550us) server_negotiation.cc:407] Connection header received
0620 09:04:26.603359 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.603363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.603440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.603490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.603985 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46060: BlockingRecv error: recv got EOF from 127.0.0.1:46060 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 09:04:26.609279 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.607773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.61:34889)
0620 09:04:26.607874 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.607877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.608367 (+ 490us) server_negotiation.cc:407] Connection header received
0620 09:04:26.608608 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.608611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.608658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.608706 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.609168 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46072: BlockingRecv error: recv got EOF from 127.0.0.1:46072 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 09:04:26.615016 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.613549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.61:34889)
0620 09:04:26.613648 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.613652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.614128 (+ 476us) server_negotiation.cc:407] Connection header received
0620 09:04:26.614323 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.614327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.614376 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.614428 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.614911 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46080: BlockingRecv error: recv got EOF from 127.0.0.1:46080 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 09:04:26.622928 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.621353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.98.61:34889)
0620 09:04:26.621499 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.621503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.621986 (+ 483us) server_negotiation.cc:407] Connection header received
0620 09:04:26.622171 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.622174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.622254 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.622307 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.622817 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46096: BlockingRecv error: recv got EOF from 127.0.0.1:46096 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 09:04:26.635142 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.633541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.98.61:34889)
0620 09:04:26.633657 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.633661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.634200 (+ 539us) server_negotiation.cc:407] Connection header received
0620 09:04:26.634406 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.634410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.634459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.634510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.635026 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46106: BlockingRecv error: recv got EOF from 127.0.0.1:46106 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 09:04:26.656049 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.654225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.98.61:34889)
0620 09:04:26.654366 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.654371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.654976 (+ 605us) server_negotiation.cc:407] Connection header received
0620 09:04:26.655213 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.655217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.655296 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.655350 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.655931 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46108: BlockingRecv error: recv got EOF from 127.0.0.1:46108 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 09:04:26.693068 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.690936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.0.98.61:34889)
0620 09:04:26.691060 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.691064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.691634 (+ 570us) server_negotiation.cc:407] Connection header received
0620 09:04:26.691938 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.691943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.691997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.692047 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.692874 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46110: BlockingRecv error: recv got EOF from 127.0.0.1:46110 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:04:26.760824 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.759677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.0.98.61:34889)
0620 09:04:26.759824 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.759829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.759997 (+ 168us) server_negotiation.cc:407] Connection header received
0620 09:04:26.760115 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.760118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.760163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.760244 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.760726 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46112: BlockingRecv error: recv got EOF from 127.0.0.1:46112 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:04:26.892237 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:26.890949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.0.98.61:34889)
0620 09:04:26.891123 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:26.891127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:26.891288 (+ 161us) server_negotiation.cc:407] Connection header received
0620 09:04:26.891486 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:26.891493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:26.891540 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:26.891625 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:26.892103 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46126: BlockingRecv error: recv got EOF from 127.0.0.1:46126 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 09:04:27.151687 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:27.150509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.98.61:34889)
0620 09:04:27.150742 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.150748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.150764 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:04:27.150954 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.150963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.151019 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.151111 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:27.151582 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46132: BlockingRecv error: recv got EOF from 127.0.0.1:46132 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 09:04:27.484553 22902 catalog_manager.cc:1259] Loaded cluster ID: 6d96fb546b4049cf81dd7ebe3314bfdb
I20260620 09:04:27.484637 22902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: loading cluster ID for follower catalog manager: success
I20260620 09:04:27.486222 22902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: acquiring CA information for follower catalog manager: success
I20260620 09:04:27.487342 22902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43a9ea312cca4d04ad64116f131ba19f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:04:27.494019 22927 catalog_manager.cc:1259] Loaded cluster ID: 6d96fb546b4049cf81dd7ebe3314bfdb
I20260620 09:04:27.494078 22927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: loading cluster ID for follower catalog manager: success
I20260620 09:04:27.495612 22927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: acquiring CA information for follower catalog manager: success
I20260620 09:04:27.496703 22927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 833d2c0e3be643f8841b1aa40600e564: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:04:27.669991 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:27.666004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.0.98.61:34889)
0620 09:04:27.666251 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.666258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.666278 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:04:27.666471 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.666474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.666591 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.666708 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:27.666982 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.667539 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.668407 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.668695 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.668828 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:27.669152 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:27.669599 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:27.669743 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:27.669778 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:04:27.669832 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260620 09:04:27.676801 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:27.673013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.0.98.60:43191)
0620 09:04:27.673262 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.673268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.673285 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:04:27.673377 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.673379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.673485 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.673609 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:27.673819 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.674383 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.675260 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.675501 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.675617 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:27.675923 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:27.676346 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:27.676510 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:27.676551 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:04:27.676636 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260620 09:04:27.684007 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:27.679681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.98.61:34889)
0620 09:04:27.679822 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.679826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.680404 (+ 578us) server_negotiation.cc:407] Connection header received
0620 09:04:27.680842 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.680845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.680906 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.680974 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:27.681295 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.681912 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.682836 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.683064 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.683188 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:27.683380 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:27.683448 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:27.683514 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:27.683792 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 09:04:27.683849 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:04:27.684173 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:27.679828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.0.98.60:43191)
0620 09:04:27.679928 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.679931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.680362 (+ 431us) server_negotiation.cc:407] Connection header received
0620 09:04:27.680592 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.680599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.680643 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.680739 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:27.681027 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.681818 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.683124 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.683438 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.683536 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:27.683723 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:27.683796 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:27.683855 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:27.683963 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:04:27.684023 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:04:27.685417 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:27.679774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.98.62:38621)
0620 09:04:27.680067 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:27.680072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:27.680273 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:04:27.680525 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:27.680531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:27.680731 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:27.680839 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:27.681265 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.682015 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.683605 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:27.683858 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:27.683946 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:27.684751 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:27.685076 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:27.685168 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:27.685238 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:04:27.685304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260620 09:04:27.695544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22661
I20260620 09:04:27.705760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22736
I20260620 09:04:27.713791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 22811
2026-06-20T09:04:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3076 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:04:27 dist-test-slave-scsf krb5kdc[23032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:04:27 dist-test-slave-scsf krb5kdc[23032](info): set up 2 sockets
Jun 20 09:04:27 dist-test-slave-scsf krb5kdc[23032](info): commencing operation
krb5kdc: starting...
W20260620 09:04:28.749069 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/test-user.keytab.
Jun 20 09:04:28 dist-test-slave-scsf krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:04:28.867074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:04:28.974200 23045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:28.974551 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:28.974627 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:28.974706 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:28.978600 23045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:28.978677 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:28.978753 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:28.978816 23045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:28.978852 23045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:28.984289 23045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:28.985397 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:28.986441 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:28.995019 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:28.995016 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:28.995014 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:28.995220 23045 server_base.cc:1034] running on GCE node
I20260620 09:04:28.995903 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:28.997045 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:28.998204 23045 hybrid_clock.cc:648] HybridClock initialized: now 1781946268998181 us; error 36 us; skew 500 ppm
Jun 20 09:04:28 dist-test-slave-scsf krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:29.000880 23045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:04:29.001775 23045 webserver.cc:460] Webserver started at http://127.0.98.62:41423/ using document root <none> and password file <none>
I20260620 09:04:29.002347 23045 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:29.002436 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:29.002662 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:29.004721 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data/instance:
uuid: "ad96cc12850c4c669ee6f2bbb249efff"
format_stamp: "Formatted at 2026-06-20 09:04:28 on dist-test-slave-scsf"
server_key: "5daa8b6bf42ff50cfdd057478e2640c1"
server_key_iv: "18a5cef45af123f2f09a78a473c771ce"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.005277 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad96cc12850c4c669ee6f2bbb249efff"
format_stamp: "Formatted at 2026-06-20 09:04:28 on dist-test-slave-scsf"
server_key: "5daa8b6bf42ff50cfdd057478e2640c1"
server_key_iv: "18a5cef45af123f2f09a78a473c771ce"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.008828 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:04:29.011246 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:29.012439 23045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:04:29.012622 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal
uuid: "ad96cc12850c4c669ee6f2bbb249efff"
format_stamp: "Formatted at 2026-06-20 09:04:28 on dist-test-slave-scsf"
server_key: "5daa8b6bf42ff50cfdd057478e2640c1"
server_key_iv: "18a5cef45af123f2f09a78a473c771ce"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.012737 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:29.027654 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:29.030539 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:29.030736 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:29.037528 23045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36957
I20260620 09:04:29.037801 23112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36957 every 8 connection(s)
I20260620 09:04:29.038470 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:04:29.041157 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:29.043367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 23045
I20260620 09:04:29.043481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-0/wal/instance
I20260620 09:04:29.043779 392 external_mini_cluster.cc:1428] Setting key 7780a141de05df26d7fa7d6da40c6aeb
I20260620 09:04:29.045466 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:29.061499 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.047857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.0.1:38506)
0620 09:04:29.048364 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.048404 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.049379 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.050061 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.050066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.050507 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.050739 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.050752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.051911 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.051914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.052743 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.052753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.053479 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.053483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.053642 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.054281 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.054303 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.055920 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.058374 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.058384 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.058396 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.058671 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.059026 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.059029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.059031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.059140 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.059726 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.059732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.059868 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.060041 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.060422 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":5,"thread_start_us":141,"threads_started":1}
I20260620 09:04:29.061499 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.048860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.98.62:36957)
0620 09:04:29.049182 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.049192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.049236 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:04:29.049680 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.049684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.049872 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.050066 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.050886 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.051783 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.052874 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.053253 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.053440 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.056091 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.056099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.056115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.056144 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.058246 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.058809 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.058822 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.058823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.058924 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.059233 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.059239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.059241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.059605 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.059743 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.060100 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.060492 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
I20260620 09:04:29.063858 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:29.064673 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.064381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39096)
0620 09:04:29.064546 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.064613 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 09:04:29.064875 23062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
I20260620 09:04:29.065702 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.065518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39100)
0620 09:04:29.065609 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.065651 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:04:29.065862 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44749: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:04:29.071473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44749
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:04:29.115154 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } attempt: 1
I20260620 09:04:29.116170 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.115791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39114)
0620 09:04:29.115957 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.116079 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 09:04:29.116886 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.116672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39118)
0620 09:04:29.116780 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.116834 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:04:29.116997 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44749: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
W20260620 09:04:29.181383 23120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:29.181700 23120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:29.181798 23120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:29.181874 23120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 09:04:29.185160 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } attempt: 2
I20260620 09:04:29.185992 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.185654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39134)
0620 09:04:29.185830 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.185902 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 09:04:29.186039 23120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:29.186084 23120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:29.186112 23120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:29.186151 23120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:29.186188 23120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:29.186724 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.186498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39138)
0620 09:04:29.186604 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.186654 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:04:29.186865 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44749: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44749: connect: Connection refused (error 111)
I20260620 09:04:29.191695 23120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:29.192804 23120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:29.193866 23120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:04:29.202087 23125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:29.202235 23120 server_base.cc:1034] running on GCE node
W20260620 09:04:29.202087 23128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:29.202069 23126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:29.202876 23120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:29.203883 23120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:29.205063 23120 hybrid_clock.cc:648] HybridClock initialized: now 1781946269205049 us; error 24 us; skew 500 ppm
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:29.207736 23120 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:04:29.208623 23120 webserver.cc:460] Webserver started at http://127.0.98.61:37709/ using document root <none> and password file <none>
I20260620 09:04:29.209211 23120 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:29.209299 23120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:29.209523 23120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:29.211445 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data/instance:
uuid: "61cdb4f974ed45b4b8fd24e69880bc81"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "717a90e07480a0e12d0eac3f66990a46"
server_key_iv: "91d0c974ccb5e913194150b0ec914703"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.211982 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal/instance:
uuid: "61cdb4f974ed45b4b8fd24e69880bc81"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "717a90e07480a0e12d0eac3f66990a46"
server_key_iv: "91d0c974ccb5e913194150b0ec914703"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.215669 23120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 09:04:29.218142 23135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:29.219296 23120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 09:04:29.219439 23120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal
uuid: "61cdb4f974ed45b4b8fd24e69880bc81"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "717a90e07480a0e12d0eac3f66990a46"
server_key_iv: "91d0c974ccb5e913194150b0ec914703"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.219544 23120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:29.235968 23120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:29.239094 23120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:29.239291 23120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:29.245986 23120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44749
I20260620 09:04:29.246246 23187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44749 every 8 connection(s)
I20260620 09:04:29.246903 23120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:04:29.247349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 23120
I20260620 09:04:29.247471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-1/wal/instance
I20260620 09:04:29.247750 392 external_mini_cluster.cc:1428] Setting key 5b50baca5eaa8acb072486154cb3206c
I20260620 09:04:29.250128 23188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:29.255728 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:29.269886 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.257915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.98.62:36957)
0620 09:04:29.258050 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.258053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.258904 (+ 851us) server_negotiation.cc:407] Connection header received
0620 09:04:29.260128 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.260132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.260195 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.260308 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.261479 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.262348 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.263515 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.263965 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.264104 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.266449 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.266452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.266455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.266482 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.267787 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.268311 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.268313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.268314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.268380 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.268800 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.268805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.268809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.269058 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.269136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.269672 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.269766 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:04:29.270840 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.258253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.0.1:38522)
0620 09:04:29.258756 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.258803 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.259917 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.260525 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.260539 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.260864 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.261337 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.261349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.262476 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.262480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.263380 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.263390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.264119 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.264121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.264256 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.264917 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.264932 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.266274 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.267886 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.267890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.267900 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.268192 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.268496 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.268499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.268501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.268696 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.269148 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.269152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.269438 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.269600 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.269888 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260620 09:04:29.272205 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:04:29.275822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Jagag/build/debug/bin/kudu
/tmp/dist-test-task3Jagag/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36355
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:04:29.278043 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } attempt: 3
I20260620 09:04:29.282456 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.272675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39140)
0620 09:04:29.272790 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.272803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.272871 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.273473 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.273480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.273610 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.273711 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.273716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.274659 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.274661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.275342 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.275353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.275937 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.275941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.276071 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.276563 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.276582 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.278226 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.280599 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.280606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.280608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.280883 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.281218 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.281222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.281224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.281338 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.281907 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.281910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.282044 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.282160 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.282288 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 09:04:29.282675 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.272790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.0.98.61:44749)
0620 09:04:29.273120 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.273129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.273147 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:04:29.273180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.273184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.273334 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.273437 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.273808 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.274565 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.275461 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.275815 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.275897 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.278372 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.278384 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.278398 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.278422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.280428 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.281002 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.281011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.281015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.281111 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.281438 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.281444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.281446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.281776 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.281932 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.282286 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.282462 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":132,"threads_started":1}
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:04:29.285198 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:29.285930 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.285663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51940)
0620 09:04:29.285790 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.285875 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 09:04:29.286145 23136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
I20260620 09:04:29.287051 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.286871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51948)
0620 09:04:29.286953 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.286995 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:04:29.287262 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
I20260620 09:04:29.288493 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.278470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39142)
0620 09:04:29.278615 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.278632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.279036 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.279356 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.279360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.279540 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.279635 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.279640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.280826 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.280830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.281886 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.281897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.282527 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.282533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.282638 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.283076 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.283096 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.284657 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.286885 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.286890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.286894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.287175 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.287513 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.287516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.287517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.287595 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.288062 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.288064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.288141 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.288219 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.288328 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 09:04:29.288712 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.278554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.98.61:44749)
0620 09:04:29.278880 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.278890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.279001 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:04:29.279138 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.279143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.279198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.279330 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.279750 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.280644 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.282040 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.282395 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.283968 (+ 1573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.284820 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.284823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.284825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.284844 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.286772 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.287316 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.287321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.287324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.287415 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.287693 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.287699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.287704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.287960 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.288074 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.288313 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.288482 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260620 09:04:29.289261 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:29.289867 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.289649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51962)
0620 09:04:29.289754 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.289802 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 09:04:29.290578 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.290326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51964)
0620 09:04:29.290470 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.290534 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 09:04:29.290717 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
I20260620 09:04:29.321858 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 1
I20260620 09:04:29.322849 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.322474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51970)
0620 09:04:29.322683 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.322750 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 09:04:29.323619 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.323372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51980)
0620 09:04:29.323509 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.323555 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 09:04:29.323771 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
I20260620 09:04:29.336603 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 1
I20260620 09:04:29.337469 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.337149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:51996)
0620 09:04:29.337334 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.337397 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 09:04:29.338196 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.337985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52006)
0620 09:04:29.338097 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.338147 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:04:29.338315 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
W20260620 09:04:29.391153 23194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:04:29.391595 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:04:29.391690 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:04:29.391765 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:04:29.396024 23194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:04:29.396129 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:04:29.396168 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:04:29.396205 23194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:04:29.396242 23194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:04:29.398931 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 2
I20260620 09:04:29.399727 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.399422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52014)
0620 09:04:29.399584 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.399648 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 09:04:29.400575 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.400327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52024)
0620 09:04:29.400424 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.400521 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:04:29.400759 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
I20260620 09:04:29.402662 23194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36957,127.0.98.61:44749,127.0.98.60:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:04:29.403699 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:04:29.404799 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:04:29.406473 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 2
I20260620 09:04:29.407203 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.406959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52030)
0620 09:04:29.407086 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.407138 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 09:04:29.407895 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.407669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52046)
0620 09:04:29.407798 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.407847 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 09:04:29.408007 23188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36355: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36355: connect: Connection refused (error 111)
W20260620 09:04:29.413412 23204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:29.413410 23202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:04:29.413467 23201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:04:29.413586 23194 server_base.cc:1034] running on GCE node
I20260620 09:04:29.414222 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:04:29.415335 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:04:29.416524 23194 hybrid_clock.cc:648] HybridClock initialized: now 1781946269416524 us; error 53 us; skew 500 ppm
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:04:29.419317 23194 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:04:29.420253 23194 webserver.cc:460] Webserver started at http://127.0.98.60:42661/ using document root <none> and password file <none>
I20260620 09:04:29.420852 23194 fs_manager.cc:359] Metadata directory not provided
I20260620 09:04:29.420938 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:04:29.421132 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:04:29.423141 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data/instance:
uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "ff36a6aecd7980a8fe90d502d64112e1"
server_key_iv: "62210768829faf12c985889b0278222a"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.423740 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal/instance:
uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "ff36a6aecd7980a8fe90d502d64112e1"
server_key_iv: "62210768829faf12c985889b0278222a"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.427356 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:04:29.429683 23211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:04:29.430944 23194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:04:29.431092 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data,/tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal
uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0"
format_stamp: "Formatted at 2026-06-20 09:04:29 on dist-test-slave-scsf"
server_key: "ff36a6aecd7980a8fe90d502d64112e1"
server_key_iv: "62210768829faf12c985889b0278222a"
server_key_version: "encryptionkey@0"
I20260620 09:04:29.431203 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:04:29.466912 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:04:29.469959 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:04:29.470166 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:04:29.477500 23194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36355
I20260620 09:04:29.477808 23263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36355 every 8 connection(s)
I20260620 09:04:29.478516 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:04:29.481434 23264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:04:29.483175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Jagag/build/debug/bin/kudu as pid 23194
I20260620 09:04:29.483301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Jagag/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781945987698834-392-0/minicluster-data/master-2/wal/instance
I20260620 09:04:29.483608 392 external_mini_cluster.cc:1428] Setting key d51c8c84e753aa82d4baff28fc6b38cb
I20260620 09:04:29.485875 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:04:29.496896 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.488044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.0.98.62:36957)
0620 09:04:29.488180 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.488184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.488483 (+ 299us) server_negotiation.cc:407] Connection header received
0620 09:04:29.488649 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.488657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.488700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.488788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.489433 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.489914 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.490875 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.491204 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.491339 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.493604 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.493608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.493612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.493642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.494980 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495441 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.495445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.495447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.495528 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495875 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.495878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.495880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.496069 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.496175 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.496593 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.496709 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:04:29.497637 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.487964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.0.1:38534)
0620 09:04:29.488349 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.488387 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.488545 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.488835 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.488840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.489193 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.489307 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.489313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.490030 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.490033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.490734 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.490743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.491365 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.491368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.491440 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.491916 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.491931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.493444 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.495099 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.495116 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.495337 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.495632 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.495638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.495756 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.496172 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.496179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.496321 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.496539 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.496867 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260620 09:04:29.498196 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.484956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.98.60:36355)
0620 09:04:29.485343 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.485368 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.485401 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:04:29.486044 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.486060 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.486439 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.486732 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.487641 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.489016 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.489868 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.490341 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.490505 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.493137 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.493143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.493156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.493185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.495003 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495461 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.495467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.495469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.495584 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.495883 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.495888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.495889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.496230 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.496451 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.496653 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.498021 (+ 1368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1160,"server-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260620 09:04:29.499166 23188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 3
I20260620 09:04:29.499415 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:29.499933 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } attempt: 3
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:04:29.509765 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.499609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52070)
0620 09:04:29.499935 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.499951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.500023 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.500556 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.500559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.500722 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.500838 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.500846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.501873 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.501878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.502746 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.502757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.503399 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.503403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.503478 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.503921 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.503935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.505916 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.507978 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.507981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.507984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.508308 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.508663 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.508668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.508761 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.509166 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.509169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.509267 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.509442 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.509568 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":22}
I20260620 09:04:29.510000 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.98.61:44749)
0620 09:04:29.500355 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.500361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.500376 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:04:29.500414 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.500417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.500488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.500559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.501149 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.501944 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.502992 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.503352 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.503422 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.505989 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.505993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.505995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.506036 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.508313 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508792 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.508798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.508803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.508870 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.509145 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.509149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.509152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.509387 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.509461 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.509759 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.509866 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":3968}
I20260620 09:04:29.510238 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.98.60:36355)
0620 09:04:29.500344 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.500349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.500362 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:04:29.500390 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.500392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.500430 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.500534 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.500969 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.501754 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.502865 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.503205 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.503303 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.506077 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.506080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.506083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.506105 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.507856 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508413 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.508417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.508418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.508557 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508857 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.508860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.508862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.509054 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.509132 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.509944 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.510041 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"spinlock_wait_cycles":3328}
I20260620 09:04:29.510967 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.0.98.62:36957)
0620 09:04:29.501138 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.501144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.501155 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:04:29.501183 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.501185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.501231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.501312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.501920 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.502658 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.503683 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.503974 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.504077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.506789 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.506793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.506797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.506825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.508653 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.509119 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.509123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.509126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.509222 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.509534 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.509538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.509541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.509803 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.510010 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.510701 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.510817 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857}
I20260620 09:04:29.511209 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44749 (local address 127.0.0.1:39152)
0620 09:04:29.500239 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.500250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.500316 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.500672 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.500675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.500893 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.501008 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.501018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.502057 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.502060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.502848 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.502858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.503552 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.503558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.503658 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.504416 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.504431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.505858 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.508413 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.508418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.508682 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.508973 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.508975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.508976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.509044 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.509501 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.509504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.509586 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.510901 (+ 1315us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.511047 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"spinlock_wait_cycles":1280}
I20260620 09:04:29.512877 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:04:29.512974 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52080)
0620 09:04:29.500606 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.500619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.500679 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.502213 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.502216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.502385 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.502476 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.502482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.504321 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.504327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.505243 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.505254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.506392 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.506398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.506517 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.507043 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.507055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.508524 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.510475 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.510478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.510480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.510713 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.511961 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.511964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.511965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.512063 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.512499 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.512505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.512601 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.512693 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.512822 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260620 09:04:29.513628 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.500908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.0.98.60:36355)
0620 09:04:29.501980 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.501991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.502004 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:04:29.502036 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.502039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.502077 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.502580 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.502653 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.504210 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.505372 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.506200 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.506391 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.508667 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.508673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.508679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.508707 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.510373 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.511543 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.511546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.511548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.511844 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.512163 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.512166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.512168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.512360 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.512484 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.513340 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.513447 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":139,"threads_started":1}
I20260620 09:04:29.514048 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: Bootstrap starting.
I20260620 09:04:29.516639 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: Bootstrap starting.
I20260620 09:04:29.516669 23188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:29.517592 23188 log.cc:824] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:04:29.519030 23113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:29.519846 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: No bootstrap required, opened a new log
I20260620 09:04:29.519870 23113 log.cc:824] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: Log is configured to *not* fsync() on all Append() calls
I20260620 09:04:29.522657 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.511981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.98.61:44749)
0620 09:04:29.512089 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.512093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.512675 (+ 582us) server_negotiation.cc:407] Connection header received
0620 09:04:29.512840 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.512843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.512883 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.512957 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.514372 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.515035 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.516116 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.516432 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.516616 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.519390 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.519393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.519396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.519420 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.521040 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.521479 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.521483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.521484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.521561 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.521844 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.521847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.521848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.522046 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.522220 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.522413 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.522519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:04:29.522684 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: No bootstrap required, opened a new log
I20260620 09:04:29.523860 23188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.524075 23188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61cdb4f974ed45b4b8fd24e69880bc81, State: Initialized, Role: FOLLOWER
I20260620 09:04:29.524402 23188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.525741 23188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:29.525837 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:04:29.526655 23113 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.526901 23113 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad96cc12850c4c669ee6f2bbb249efff, State: Initialized, Role: FOLLOWER
I20260620 09:04:29.526919 23087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ad96cc12850c4c669ee6f2bbb249efff"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38550
I20260620 09:04:29.527222 23113 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.527487 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.527690 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.528035 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.528264 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 09:04:29 dist-test-slave-scsf krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:04:29.528551 23113 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:29.528594 23087 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:04:29.528697 23087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:29.530055 23087 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.531975 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad96cc12850c4c669ee6f2bbb249efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61cdb4f974ed45b4b8fd24e69880bc81"
I20260620 09:04:29.532191 23162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:29.532522 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.513440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.0.98.60:36355)
0620 09:04:29.513517 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.513521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.513531 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:04:29.513559 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.513561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.513593 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.513667 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.520598 (+ 6931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.521299 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.525542 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.525836 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.525908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.529089 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.529092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.529095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.529115 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.530796 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.531311 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.531314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.531316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.531382 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.531625 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.531628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.531629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.531815 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.531889 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.532234 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.532351 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:04:29.533135 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.513286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36355 (local address 127.0.0.1:52092)
0620 09:04:29.513372 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 09:04:29.513384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:04:29.513448 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:04:29.514012 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:04:29.514015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:04:29.520315 (+ 6300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:04:29.520468 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.520478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.524559 (+ 4081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.524653 (+ 94us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.525408 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:04:29.525419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.526096 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.526099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:04:29.526172 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.526649 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:04:29.526663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:04:29.528970 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:04:29.530991 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.530994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.530996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.531222 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.531468 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:04:29.531471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:04:29.531475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:04:29.531553 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:04:29.531928 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:04:29.531931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:04:29.532016 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 09:04:29.532864 (+ 848us) client_negotiation.cc:240] Negotiation successful
0620 09:04:29.532995 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:04:29.533509 23162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad96cc12850c4c669ee6f2bbb249efff in term 1.
I20260620 09:04:29.534169 23062 leader_election.cc:302] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61cdb4f974ed45b4b8fd24e69880bc81, ad96cc12850c4c669ee6f2bbb249efff; no voters:
W20260620 09:04:29.534197 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:29.534277 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:29.534417 23278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:04:29.534610 23278 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [term 1 LEADER]: Becoming Leader. State: Replica: ad96cc12850c4c669ee6f2bbb249efff, State: Running, Role: LEADER
I20260620 09:04:29.534907 23278 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.536640 23264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: Bootstrap starting.
I20260620 09:04:29.531021 23087 leader_election.cc:288] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [CANDIDATE]: Term 1 election: Requested vote from peers 61cdb4f974ed45b4b8fd24e69880bc81 (127.0.98.61:44749), fffaaf2a7d2c43e9bf5e4d4e61805ee0 (127.0.98.60:36355)
I20260620 09:04:29.536895 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad96cc12850c4c669ee6f2bbb249efff. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.536993 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:29.539033 23264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: Neither blocks nor log segments found. Creating new log.
I20260620 09:04:29.529704 23120 master_runner.cc:186] Error getting master registration for 127.0.98.62:36957: OK, instance_id { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" instance_seqno: 1781946269035636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:04:29.539384 23120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:04:29.539885 23264 log.cc:824] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: Log is configured to *not* fsync() on all Append() calls
W20260620 09:04:29.532650 23066 tablet.cc:2369] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:04:29.543265 23284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:29.543334 23284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:29.543644 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:04:29.544595 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:04:29.542510 23045 master_runner.cc:186] Error getting master registration for 127.0.98.60:36355: OK, instance_id { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" instance_seqno: 1781946269475380 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:04:29.546901 23045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:04:29.550609 23162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [term 1 FOLLOWER]: Refusing update from remote peer ad96cc12850c4c669ee6f2bbb249efff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:04:29.551265 23297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [LEADER]: Connected to new peer: Peer: permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:04:29.555627 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad96cc12850c4c669ee6f2bbb249efff. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.555795 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.557610 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.557732 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:29.557829 23309 mvcc.cc:205] Tried to move back new op lower bound from 7298851920072794112 to 7298851920018530304. Current Snapshot: MvccSnapshot[applied={T|T < 7298851920072794112}]
I20260620 09:04:29.558141 23310 mvcc.cc:205] Tried to move back new op lower bound from 7298851920072794112 to 7298851920018530304. Current Snapshot: MvccSnapshot[applied={T|T < 7298851920072794112}]
I20260620 09:04:29.558600 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.558717 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.560483 23299 catalog_manager.cc:1347] Generated new cluster ID: 76409631a2634b088ffcf603f50490d9
I20260620 09:04:29.560547 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:04:29.561604 23264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: No bootstrap required, opened a new log
I20260620 09:04:29.565802 23264 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.566042 23264 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffaaf2a7d2c43e9bf5e4d4e61805ee0, State: Initialized, Role: FOLLOWER
I20260620 09:04:29.566365 23264 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } }
I20260620 09:04:29.567687 23264 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:04:29.568195 23238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad96cc12850c4c669ee6f2bbb249efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0"
I20260620 09:04:29.568408 23238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:04:29.569816 23238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad96cc12850c4c669ee6f2bbb249efff in term 1.
I20260620 09:04:29.570437 23237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [term 1 FOLLOWER]: Refusing update from remote peer ad96cc12850c4c669ee6f2bbb249efff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:04:29.570960 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.571086 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.571239 23297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:04:29.573916 23325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:04:29.576066 23325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:04:29.578099 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad96cc12850c4c669ee6f2bbb249efff. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.578235 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.578338 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.578416 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:04:29.586526 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.545675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.98.61:44749)
0620 09:04:29.545793 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.545798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.565091 (+ 19293us) server_negotiation.cc:407] Connection header received
0620 09:04:29.565266 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.565269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.565318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.565369 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.567294 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.568040 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.573866 (+ 5826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.574165 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.574240 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.578088 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.578092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.578095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.578122 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.580031 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.583898 (+ 3867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.583902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.583903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.583980 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.584595 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.584600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.584601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.584852 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.584970 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.586256 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.586377 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:04:29.588284 23327 mvcc.cc:205] Tried to move back new op lower bound from 7298851920072794112 to 7298851920018530304. Current Snapshot: MvccSnapshot[applied={T|T < 7298851920072794112}]
I20260620 09:04:29.588431 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.544934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.0.98.62:36957)
0620 09:04:29.545044 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.545048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.564800 (+ 19752us) server_negotiation.cc:407] Connection header received
0620 09:04:29.564911 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.564914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.564961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.565032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.571280 (+ 6248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.571942 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.574892 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.575203 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.575280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.581947 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.581951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.581953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.581980 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.583430 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.584217 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.584220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.584222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.584290 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.585586 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.585590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.585591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.585822 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.585925 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.588148 (+ 2223us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.588278 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:04:29.591071 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:29.539015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.0.98.60:36355)
0620 09:04:29.539121 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.539126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.556774 (+ 17648us) server_negotiation.cc:407] Connection header received
0620 09:04:29.556960 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.556963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.557024 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.557106 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.558343 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.559123 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.568547 (+ 9424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:29.568860 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:29.568934 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:29.580171 (+ 11237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:04:29.580175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:04:29.580178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:04:29.580207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:04:29.581784 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.589851 (+ 8067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.589854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.589856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.589939 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:04:29.590260 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:04:29.590263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:04:29.590265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:04:29.590443 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:04:29.590543 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:29.590834 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 09:04:29.590936 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:04:29.595269 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad96cc12850c4c669ee6f2bbb249efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad96cc12850c4c669ee6f2bbb249efff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } peers { permanent_uuid: "61cdb4f974ed45b4b8fd24e69880bc81" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44749 } } peers { permanent_uuid: "fffaaf2a7d2c43e9bf5e4d4e61805ee0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36355 } } }
I20260620 09:04:29.595391 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff [sys.catalog]: This master's current role is: LEADER
I20260620 09:04:29.600538 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:04:29.601859 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:04:29.617177 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad96cc12850c4c669ee6f2bbb249efff: Generated new TSK 0
I20260620 09:04:29.618134 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:04:29.647330 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.644303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.0.98.62:36957)
0620 09:04:29.644445 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.644450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.646230 (+ 1780us) server_negotiation.cc:407] Connection header received
0620 09:04:29.646428 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.646431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.646480 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.646534 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.647180 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38570: BlockingRecv error: recv got EOF from 127.0.0.1:38570 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 09:04:29.648672 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.645036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.98.61:44749)
0620 09:04:29.645147 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.645152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.647781 (+ 2629us) server_negotiation.cc:407] Connection header received
0620 09:04:29.647955 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.647958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.648014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.648058 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.648540 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39180: BlockingRecv error: recv got EOF from 127.0.0.1:39180 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 09:04:29.650091 23267 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.645672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.0.98.60:36355)
0620 09:04:29.645772 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.645776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.649038 (+ 3262us) server_negotiation.cc:407] Connection header received
0620 09:04:29.649392 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.649396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.649444 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.649493 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.649978 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52106: BlockingRecv error: recv got EOF from 127.0.0.1:52106 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 09:04:29.654177 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.652806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.0.98.61:44749)
0620 09:04:29.652901 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.652905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.653314 (+ 409us) server_negotiation.cc:407] Connection header received
0620 09:04:29.653516 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.653519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.653594 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.653642 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.654087 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39194: BlockingRecv error: recv got EOF from 127.0.0.1:39194 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 09:04:29.658941 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.657606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.0.98.61:44749)
0620 09:04:29.657690 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.657693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.658133 (+ 440us) server_negotiation.cc:407] Connection header received
0620 09:04:29.658316 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.658319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.658362 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.658408 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.658857 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39200: BlockingRecv error: recv got EOF from 127.0.0.1:39200 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 09:04:29.664216 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.662848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.98.61:44749)
0620 09:04:29.662938 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.662942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.663404 (+ 462us) server_negotiation.cc:407] Connection header received
0620 09:04:29.663572 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.663576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.663619 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.663664 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.664115 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39214: BlockingRecv error: recv got EOF from 127.0.0.1:39214 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 09:04:29.671674 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.670277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.98.61:44749)
0620 09:04:29.670401 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.670405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.670855 (+ 450us) server_negotiation.cc:407] Connection header received
0620 09:04:29.671038 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.671041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.671081 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.671124 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.671590 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39216: BlockingRecv error: recv got EOF from 127.0.0.1:39216 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 09:04:29.683267 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.681800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.98.61:44749)
0620 09:04:29.681903 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.681907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.682385 (+ 478us) server_negotiation.cc:407] Connection header received
0620 09:04:29.682562 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.682565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.682612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.682658 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.683131 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39228: BlockingRecv error: recv got EOF from 127.0.0.1:39228 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 09:04:29.703567 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.701876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.0.98.61:44749)
0620 09:04:29.701997 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.702002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.702564 (+ 562us) server_negotiation.cc:407] Connection header received
0620 09:04:29.702764 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.702768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.702825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.702879 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.703443 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39230: BlockingRecv error: recv got EOF from 127.0.0.1:39230 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 09:04:29.740118 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.738360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.98.61:44749)
0620 09:04:29.738496 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.738502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.739110 (+ 608us) server_negotiation.cc:407] Connection header received
0620 09:04:29.739327 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.739331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.739383 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.739437 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.739999 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39232: BlockingRecv error: recv got EOF from 127.0.0.1:39232 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 09:04:29.808568 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.807375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.0.98.61:44749)
0620 09:04:29.807518 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.807523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.807696 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:04:29.807872 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.807875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.807918 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.808002 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.808418 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39244: BlockingRecv error: recv got EOF from 127.0.0.1:39244 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 09:04:29.939867 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:29.938607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.0.98.61:44749)
0620 09:04:29.938833 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:29.938838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:29.938972 (+ 134us) server_negotiation.cc:407] Connection header received
0620 09:04:29.939151 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:29.939154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:29.939211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:29.939305 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:29.939754 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39260: BlockingRecv error: recv got EOF from 127.0.0.1:39260 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 09:04:30.199198 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:04:30.198069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.0.98.61:44749)
0620 09:04:30.198234 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.198240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.198258 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:04:30.198435 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.198438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.198493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.198584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:04:30.199089 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39266: BlockingRecv error: recv got EOF from 127.0.0.1:39266 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:04:30.544647 23284 catalog_manager.cc:1259] Loaded cluster ID: 76409631a2634b088ffcf603f50490d9
I20260620 09:04:30.544745 23284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: loading cluster ID for follower catalog manager: success
I20260620 09:04:30.546489 23284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: acquiring CA information for follower catalog manager: success
I20260620 09:04:30.547684 23284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61cdb4f974ed45b4b8fd24e69880bc81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:04:30.577427 23325 catalog_manager.cc:1259] Loaded cluster ID: 76409631a2634b088ffcf603f50490d9
I20260620 09:04:30.577513 23325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: loading cluster ID for follower catalog manager: success
I20260620 09:04:30.579159 23325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: acquiring CA information for follower catalog manager: success
I20260620 09:04:30.580271 23325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fffaaf2a7d2c43e9bf5e4d4e61805ee0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:04:30.717144 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:30.713422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.0.98.61:44749)
0620 09:04:30.713664 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.713667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.713682 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:04:30.713828 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.713831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.713953 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.714054 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:30.714271 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.714791 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.715619 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.715864 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.715942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:30.716256 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:30.716749 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:30.716851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:30.716896 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 09:04:30.716941 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260620 09:04:30.723584 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:30.720110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.0.98.60:36355)
0620 09:04:30.720328 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.720331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.720343 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:04:30.720425 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.720428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.720591 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.720686 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:30.720878 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.721378 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.722122 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.722325 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.722404 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:30.722754 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:30.723215 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:30.723344 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:30.723383 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:04:30.723433 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260620 09:04:30.732370 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:30.726799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.0.98.60:36355)
0620 09:04:30.727668 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.727674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.727690 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:04:30.727740 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.727746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.727810 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.727898 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:30.728407 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.729331 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.730915 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.731351 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.731480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:30.731871 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:30.731971 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:30.732041 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:30.732190 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:04:30.732242 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 09:04:30.736393 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:30.731806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.0.98.61:44749)
0620 09:04:30.731932 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.731938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.732733 (+ 795us) server_negotiation.cc:407] Connection header received
0620 09:04:30.732971 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.732974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.733026 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.733075 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:30.733414 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.734257 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.735143 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.735534 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.735650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:30.735891 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:30.735964 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:30.736047 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:30.736180 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 09:04:30.736232 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:04:30.737653 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:04:30.726457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.0.98.62:36957)
0620 09:04:30.726704 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 09:04:30.726709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:04:30.732432 (+ 5723us) server_negotiation.cc:407] Connection header received
0620 09:04:30.732622 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:04:30.732626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:04:30.732795 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:04:30.733140 (+ 345us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:04:30.733251 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.734120 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.735589 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:04:30.736043 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:04:30.736167 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:04:30.736438 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:04:30.736927 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:04:30.737029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:04:30.737368 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 09:04:30.737444 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260620 09:04:30.741068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 23045
I20260620 09:04:30.747512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 23120
I20260620 09:04:30.753499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Jagag/build/debug/bin/kudu with pid 23194
2026-06-20T09:04:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3037 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6114 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283044 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST